A customer-service AI is asked how long the warranty runs. It answers in three seconds, sounds certain, and cites its source page. The answer is wrong — the page it cited describes the previous model's terms. Here's the uncomfortable part: nobody wrote that page. The system wrote it, filing away its own answer from a question someone asked last month. The error now looks exactly like a correct answer — titled, structured, cross-linked, indistinguishable from a real contract clause when cited.
First, what this self-writing knowledge base actually does
The design has been circulating fast among engineers. It traces to a technical note Andrej Karpathy posted this April, and several vendors have since packaged it into enterprise products. The idea fits in one sentence: rather than have the AI dig through raw documents every time somebody asks a question, have it behave like a wiki editor who never goes home — compiling what it reads into an internal encyclopedia that gets thicker over time, and answering from that instead.
Unpacked, it's three layers and three loops. The layers: raw source documents that never change (contracts, filings, meeting notes); a wiki of structured, cross-linked pages the LLM maintains; and a schema governing how that wiki is organized and how each kind of page should be written. The loops: Ingest reads new material in and compiles it into pages, Query answers questions from the wiki, and Lint periodically scans the whole thing for contradictions and stale content.
The real ingenuity sits inside Query, in a move called file-back: once the AI answers a question well, that answer is written back into the wiki as a new page — and becomes the first thing read the next time someone asks something similar. The more it's used, the thicker the wiki, the faster the next answer. The customer-service error above grew out of exactly this.
Why it excites people: knowledge settles instead of evaporating
To see why this lands, look at what it's arguing against. Ordinary RAG retrieval works like this: a user asks, the system grabs the few most relevant passages from a document pile, hands them to the model, the model assembles an answer — and then nothing. The work of assembling that answer, the effort of stitching a conclusion out of ten documents, evaporates the moment it's delivered. Ask the same question tomorrow and the whole thing runs again from zero. A hundred queries, a hundred fresh starts.
The wiki approach keeps that work. The answer goes to the user and settles back into the knowledge base as an asset. Vendors call this knowledge compounding — every use makes the knowledge base more valuable instead of leaving it where it was. It's an attractive claim, and it's why we're paying attention to this line of work ourselves. The honest caveat belongs up front: in the public discussion so far, compounding is an analogy, and I haven't seen longitudinal data long enough to support that elegant rising curve. The direction is reasonable, but right now it is a promising hypothesis rather than a verified result — a distinction that matters a great deal when you're the one signing the budget.
The same file-back cannot tell what it is writing back
Grant the compounding claim, and the question that actually matters arrives: when the answer being written back is wrong, what exactly is compounding?
The write-ups productizing this architecture usually do name the risk, in words close to “the error gets encyclopedia-ized” — authoritative-looking and wrong. Having named it, they move on to the product's other virtues. What none of them explains is what happens next: once that page is filed back, who catches that it's wrong, and when?
The standard answer is the Lint loop — have the LLM periodically sweep the wiki for contradictions and stale pages. Reasonable, until you notice one thing: the model running Lint is usually the same model that wrote the page.
Asking a model to catch its own errors is asking it to see its own blind spot
If a model errs in a particular way — swapping clauses between two similarly formatted contracts, writing “warranty” where it meant “coverage” and dropping the difference in scope — the judgment it used to write that error during Ingest is drawn from the same training distribution as the judgment it brings to Lint. What it catches is precisely the class of mistake it wasn't prone to making. What it misses is its own systematic blind spot.
LLMs grading their own output is well-trodden research ground, and the finding is consistent: self-consistency is not correctness. It only means the model gives you the same answer to the same question on two different days. So in a wiki written and reviewed by one model, the most dangerous content isn't the obviously absurd error — those tend to collide with some other signal and get caught. It's the page the model finds plausible enough that it never flags it at all. Those sit quietly in the knowledge base, waiting to be cited.
The other half of the ledger: the wiki grows faster than humans can review
So a human has to step in. That is every enterprise version's answer: add a review gate — AI drafts, a person signs off. The trouble is that this defense doesn't survive its own arithmetic.
In the compounding story, usage is good news: more questions, more file-backs, a faster-growing wiki. Flip that same curve over and it's review load — Ingest updating a dozen pages a batch, every good Query answer potentially spawning another, Lint producing a fresh contradiction list on schedule. All of it lands on the same group of people, and that group is usually the same size on day 100 as it was on day one.
We've watched the script that follows too many times. Week one: everything gets read carefully. Week two: selective review, only the pages that look important. Month one: green checkmark, approved. That isn't a failure of discipline. It's a failure of interface. Reading a diff and deciding whether this particular edit deserves trust is cognitively expensive work, and what most systems hand the reviewer is a raw diff with nothing to help them rank which pages are worth stopping on. Writing “human review” into a flowchart does not make it happen in the world.
So whenever we look at a workflow where AI drafts and a person signs off, the first question is always the same: does the review curve grow with the output curve? Where it doesn't, the month the compounding story hits a wall is something you can calculate in advance.
Swap the model, and nobody can tell whose judgment is on the old pages
There's a further layer almost nobody discusses. A wiki is edited by a model, and editing carries a point of view: what's important enough to deserve its own page, what's minor enough to fold into someone else's footnote, which of two conflicting claims to believe. Those are judgments, not transcription.
Swap the underlying model and those judgments shift wholesale. The new model's bar for “important,” its sensitivity to conflict, even its idea of “concise” will not replicate its predecessor's. A wiki spanning two or three model generations therefore stacks up more than knowledge — it stacks up several editorial sensibilities that disagree with each other, and typically not one page is stamped with which model wrote it, under which version of the schema.
The day you find an old page whose reasoning doesn't square with how the system writes today, you'll have no way to trace it. You'll be guessing. A knowledge asset built for genuine compounding has to record more than what changed on a page — it has to record whose judgment made the call: which model, which schema version, signed off by whom. Neither the original technical note nor the commercial versions built on top of it address this.
Three questions to ask before adopting one
Before committing to any “AI drafts, a human signs off” knowledge system — built in-house or bought from a vendor — three questions are worth asking to someone's face. First: when a page turns out wrong, how does it get reverted or corrected — is there a record, or does it just get silently overwritten? Second: is reviewer headcount budgeted to grow with usage, or is “human review” doing all its work as three words on a slide? Third: does every page record which model version and which schema version produced it, so that after the next model upgrade you can tell which judgments deserve a second look?
Back to the customer-service AI and its wrong warranty. If every page it cited carried an audit trail, every edit traced to whoever signed it off, and every page was stamped with the model generation that wrote it, then getting the answer wrong becomes an ordinary event — traceable, correctable, maybe even the input to the next Lint pass. Strip those three things out and what's left is a guessing machine dressed up as an encyclopedia.
For knowledge to compound, what compounds has to be a verified judgment, not a formatted one. Without an audit trail, the thing growing fastest isn't your asset. It's the error nobody has caught yet.
