RAG Is Context Engineering, Not a Chatbot Trick

Retrieval isn’t the vector database. It’s the discipline of not loading everything.

Ask ten engineers to explain RAG (retrieval-augmented generation) and nine of them will draw you the same three-step diagram: embed your documents, run a similarity search, stuff the top handful of chunks into the prompt. For a chatbot answering questions against a static PDF, that diagram is basically the whole story, and it’s a genuinely useful trick. It’s also, I’ve come to think, exactly why so many people misjudge what retrieval is actually doing once you stop asking one question and start running something that works for hours: an agent holding a task in its head across dozens of steps, reading code, editing files, checking its own work, coming back to it the next day. Retrieval doesn’t stop mattering once the interaction isn’t one-shot anymore. It becomes the whole discipline.

The three-step diagram, and where it runs out of road

The naive loop earns its keep for exactly the problem it was built for: a user asks a question, you search a document store for the passages that answer it, you hand those passages to the model, done. One query, one answer, context window closed the moment the response finishes.

An agent doesn’t work that way. It has a context window that behaves like RAM, not a hard drive: fixed, finite, and everything not currently in it might as well not exist. Across a long task, that window fills with the system prompt, the running transcript, every tool result, every file it’s read. Bolt a naive RAG step onto that setup and you’ve just added one more static blob to a window already under pressure: search once, dump the results in, and now they sit there for the rest of the run whether or not step forty still needs what step four fetched. That’s the mechanism behind what I’ve called context rot elsewhere: memory pressure for language models, the good instruction from ten turns ago quietly buried under everything piled on top of it since.

The diagram never says what to do about that, because it assumes retrieval happens once, before the task starts. For an agent, the task is the retrieval problem, over and over, at every step.

What retrieval is actually managing

Reframe it and the picture gets simpler, not more complicated. For a long-running agent, retrieval isn’t a document search bolted onto the front of a conversation. It’s the mechanism deciding, at every step, what slice of a much bigger world (memory, code, conventions, prior decisions) is worth pulling into a limited window right now, and what gets left outside it. The win was never “give the model access to everything.” A model with access to everything and no discipline about it is worse off than one with access to the right five things, because the wrong four hundred are actively burying the signal.

That’s a design problem before it’s a database problem. Whether the retrieval mechanism is a vector index, a hand-written summary, or a grep, the job is the same: hold the smallest correct slice of context for the step in front of you, and go get more the moment you actually need it.

The plainest version: an index, not a dump

The simplest form of this I run day to day has nothing to do with embeddings. Every agent I run keeps a small, persistent memory: durable facts about a project, standing feedback on how I like work done, gotchas worth not relearning. Stored the naive way, that memory would just accumulate, and every session would open by reading the whole pile before doing anything else, which is exactly the load-everything failure the three-step diagram encourages.

Instead it’s structured as an index. One file holds a single line per memory: a short, specific description of what that memory covers and why it matters, each pointing at its own separate file. A session starts by reading the index, not the archive. From a dozen one-line descriptions, it decides which two or three are actually relevant to the task in front of it, opens only those, and leaves the rest alone. Nothing forces every fact I’ve ever recorded about a project into every conversation about that project. The index is the retrieval step, and the discipline is entirely in keeping that index honest: specific enough that a one-line description tells you whether to bother opening the file, current enough that it isn’t pointing at something three sessions out of date.

That’s RAG with the fancy part removed. No embeddings, no vector store, just a curated index and a rule about reading it before reading anything it points to. It works precisely because it’s small enough for a human to keep honest.

When the fancy part earns its keep

Which is also the limit of it. A hand-curated index scales to a corpus a person can actually maintain: dozens of memories, hundreds of files, something you could review in an afternoon. Past that, the index itself becomes the thing decaying: unpruned entries, stale summaries, descriptions that no longer match what’s in the file behind them. That’s exactly where semantic retrieval, real embeddings and a real vector search, starts paying for itself: a corpus too large for anyone to hand-index, queries phrased in whatever words a user happens to reach for rather than the vocabulary the index was written in, recall that matters more than precision because you’d rather retrieve six chunks and let the model sort out the right one than miss it entirely.

The two aren’t competing techniques. They’re the same idea, retrieve the slice instead of the whole, at two different scales. A hand-written index wins on precision and auditability when the corpus is small enough for a person to own. Embeddings win on recall and coverage once it isn’t. Picking between them is a question about the size and shape of what you’re retrieving from, not a question about which one counts as “real” RAG.

Retrieval over structure, not just prose

Neither of those is limited to text, and this is the part the chatbot framing misses hardest. Before an agent touches a function that six other files call, the retrieval step that actually matters isn’t a search over prose notes. It’s a query against the code’s real structure: who calls this, what breaks if the signature changes, which tests exercise the path. That’s retrieval against a call graph instead of a document store, and it’s frequently worth more than any similarity search, because a dependency graph doesn’t hallucinate a plausible-sounding but wrong answer the way a fuzzy semantic match sometimes can. Ask the code where its edges are before you cut one.

I lean on the same instinct in the small CLIs I use to run a whole GitHub estate: one of them exists purely to answer “what shipped, and when,” and it answers that by querying real git history and the GitHub API directly, not a database someone has to remember to keep in sync with reality. Retrieval over the source of truth beats retrieval over a copy of it, every time the copy has a chance to go stale.

The same discipline shows up one level above the code, in which rules even apply to an agent in the first place. A machine-wide policy, an estate-wide layer, a project’s own file: every session automatically assembles the slice of governance that actually applies to the repo it’s sitting in, without ever loading a rule written for some other project it isn’t touching. That’s retrieval too. Not “search my notes for something relevant,” but “assemble exactly the layered subset of policy that governs this task, in this place, right now,” the same instinct as the memory index, aimed at conventions instead of facts.

The discipline, named plainly

Strip the mechanism away, embeddings or a hand-written index, prose or a code graph, memory or policy, and one rule holds across every version of it: a focused context beats a big one. Retrieve the slice the current step actually needs. Prune what the last step needed and this one doesn’t. Trust the agent to ask for more the moment it discovers it’s missing something, rather than trying to guess everything it might ever need and front-loading all of it. That’s the actual lesson underneath the term RAG, and it has very little to do with whether there’s a vector database involved.

It has an honest failure mode too, worth naming rather than glossing over. A retrieval system nobody curates turns into exactly the pile it was built to prevent: stale entries nobody prunes, an index padded with descriptions vague enough to match everything, a top-k search that returns five plausible chunks while the one that actually mattered was the sixth. Retrieval isn’t a fire-and-forget fix you bolt onto an agent once. Somebody still has to own whether the index is honest, whether the memory written down last month still deserves to be there, whether “relevant” is being decided by anything more careful than a keyword hitting a description. Retrieval reduces what an agent has to hold in its head at once. It doesn’t remove the judgment about what belongs in that reduced set. It just moves that judgment somewhere you can actually inspect it.

Where that leaves the diagram

The three-step version isn’t wrong. It’s describing one instance of a much bigger idea, sized for the smallest case: one question, one search, one answer. The moment the interaction stretches past a single turn, retrieval stops being a step you run once at the start and becomes the ongoing answer to a question you’re asking constantly: given everything this task could plausibly need, what actually belongs in the window right now?

That’s not a database problem. It’s a discipline problem, and no amount of embeddings substitutes for actually solving it. The vector database is optional. Deciding what belongs in the room right now never is.