The missing layer in every data platform

Every data team has had this conversation. Someone is asked to change a nightly load, a materialised view or a calculation that feeds a finance report. They open the code. The code is clear about what happens. It says nothing about why. There is a filter that looks wrong, a date shifted by four hours, a value multiplied back up after being divided down. The documentation, if there is any, describes a version of the system from several releases ago.

So they do what everyone does. They ask the person who has been there longest. If that person is available, the change takes a day. If not, it waits, or it goes ahead on a best guess.

That person is not a process. They are a single point of failure that most organisations have quietly accepted for years. This article is about why that arrangement is breaking, why the usual fixes have not worked, and what we think should replace it.

What institutional knowledge actually is

“Institutional knowledge” gets used loosely, so it is worth being precise. For a data platform, the knowledge a team relies on comes in three layers:

• What the system does. Which procedures run, in which order, reading which tables and writing which others.
• How it is operated. How a load is restarted after a failure, how a new market is onboarded, which steps must never run in parallel.
• Why it is the way it is. The contractual clause behind a cut-off time, the reporting rule behind a filter, the incident that explains an odd workaround.

The code captures the first layer completely and the second layer partially. It almost never captures the third. Yet the third layer is what decides whether a change is safe. A filter that looks like a bug and a filter that implements a disclosure rule are indistinguishable in SQL.

Institutional knowledge, in practice, is the how and the why: the part of a system's design that lives in people, tickets, chat threads and contracts rather than in the system itself.

Why data platforms are hit hardest

Every software organisation has some of this problem. Data platforms have more of it, for structural reasons.

• They live a long time. A warehouse built fifteen years ago is often still running, extended in layers by people who have since moved on. Application code gets rewritten; warehouse code accumulates.
• They encode the business, not just the software. Fiscal calendars, revenue recognition, commission treatment and regulatory definitions all end up as SQL. The reasons for them sit in finance, legal and operations, not in engineering.
• Their failures are silent. A broken application throws an error. A broken data pipeline usually produces a plausible number that is wrong, and nobody notices until a report is questioned.
• The teams are small relative to the surface. A handful of engineers maintain thousands of objects. The ratio of knowledge to people is high, so each departure removes a lot of it.

Why the usual fixes don't hold

None of this is new, and organisations have tried to fix it. The attempts tend to fall into four groups. Each helps. None closes the gap, and it is worth understanding why.

Documentation drives

A quarter is set aside, a wiki space is created, and engineers write down what they know. It is valuable on the day it is written. Then the system changes and the pages do not. Within a year nobody can tell which pages are still true, and a wrong page is worse than a missing one because people trust it. The underlying issue is that a wiki has no relationship with the system it describes. It cannot notice when it has become wrong.

Handover sessions

When someone leaves, they walk a colleague through what they know. This captures what the departing person remembers to mention, in a format nobody else can search, at the moment when they are least invested in it. It is better than nothing and much worse than it feels.

Enterprise search and AI assistants over documents

Search tools make existing documents easier to find, and AI assistants summarise them fluently. But they inherit every problem of the documents underneath. If the page is out of date, the answer is out of date, now delivered with more confidence. And they cannot find what was never written down, which for the why layer is most of it.

Code assistants

Tools that read the codebase are excellent at the what layer. They can explain a procedure, trace a call chain and suggest a change. What they cannot do is see intent that is not in the code. Asked why a filter exists, they infer an answer from naming and structure. Sometimes the inference is right. When it is wrong, it is indistinguishable from when it is right.

The common failure is the same in all four. None of them checks what it says against the system, and none of them knows what it does not know.

Why it matters more now

For most of the last twenty years, missing institutional knowledge was a cost: slower changes, longer onboarding, occasional incidents. Teams absorbed it because a senior engineer was always in the loop to catch the dangerous changes.

That assumption is ending. Organisations are starting to let AI write SQL, propose pipeline changes and, in some cases, make those changes directly. An AI agent has access to the schema, the code and whatever documents it is given. It does not have access to the conversation in which a team decided that the comparison base is thirteen periods rather than twelve months. Given an outdated wiki page that says twelve months, it will make the “correction” and explain its reasoning clearly.

The constraint on delegating work to AI in data platforms is not the capability of the models. It is that the knowledge needed to act safely is not anywhere the model can read it. Human review becomes the bottleneck, and the reviewers are the same few people who were already the bottleneck.

Defining the category: verified knowledge

We think the answer is a distinct layer in the data stack, alongside the catalogue, the orchestrator and the observability tools. We call it verified knowledge.

Verified knowledge is a maintained record of how a system works and why, in which every statement is attributed to its source, checked against the system itself, and explicit about what is not yet known.

Five properties separate it from documentation and from search:

1. Captured from the source. Knowledge comes from the people who made the decisions and the documents they point to, and every statement is attributed. Who said it, and where, is part of the record.
2. Checked against the system. Each statement about behaviour is compared with the code it refers to and marked confirmed, contradicted or unverified. When a document and the code disagree, the code decides, and the document is flagged.
3. Honest about gaps. What nobody has recorded is visible as an open question with an owner, not hidden behind a fluent answer. Coverage is something you can measure.
4. Kept current. When a team changes how something works, the old knowledge is retired rather than left alongside the new, and contradictions surface as soon as they appear.
5. Readable by machines. The record is structured so that AI agents can query it and receive the same sourced, verified answers a person would.

Of these, the second is the one that matters most and the one existing tools lack. A knowledge base that cannot tell whether it is still true will decay, however well it is written. Tying each statement to the code it describes is what lets the record stay trustworthy as the system changes.

What this looks like in practice

This is the approach we have taken with Telvi.

• Start from the code. Telvi indexes the codebase first, so it knows what exists and what is undocumented. That tells it what to ask, and whom.
• Ask the people who know. Engineers receive short, specific questions about the parts of the system they own. They answer in their own words and can point to a ticket or a page. Only what they, or the linked page, actually say is recorded.
• Let the code settle it. Each statement is checked against the code it names. Agreement is recorded as confirmed; disagreement opens a question rather than silently overwriting either side.
• Answer from the record, not from the internet. When someone asks a question, Telvi answers from what has been captured and verified, cites each source, and says plainly where the record stops. Questions it cannot answer become open questions for the right person, so the record grows where it is needed.

The result is not a document. It is a graph of statements, each with its author, its evidence, its status against the code and its history, that both people and agents can query.

Where to start

You do not need a platform to begin. The practice matters more than the tooling, and it is worth trying on a small scale:

1. Pick one area of the platform where one or two people are the bottleneck.
2. List the questions they are asked most often. Ten is enough.
3. Record their answers in their words, with a link to any ticket or page they cite.
4. For each answer, find the lines of code that confirm or contradict it, and note which.
5. Keep a separate list of the questions nobody could answer, and give each one an owner.

Done by hand, this produces a useful record within a few weeks. It also shows why the work needs tooling: the verification step is the expensive part, and it has to be repeated every time the code changes. That is the part Telvi automates.

If you would like to see what this looks like on your own platform, we run pilots on a single area at a time.