What is RAG, really?

RAG, retrieval-augmented generation, is the technique that lets a language model answer from documents it was never trained on. Instead of relying on what the model memorized, you fetch the most relevant passages from your own data at the moment of the question and put them in the prompt alongside it. The model still writes the answer; the retrieval step decides what it gets to read first.

That one idea, supplying knowledge at query time instead of baking it into the weights, was introduced as retrieval-augmented generation in a 2020 paper and is now the default way to put a model on top of private or fast-changing data. This piece builds on two earlier ones: the embeddings explainer, which covers how text becomes comparable vectors, and the context window explainer, which covers the space the retrieved text has to fit into.

What RAG is doing, step by step

The whole pattern is six steps, and they split cleanly into two passes. The first pass happens once, ahead of any question. The second runs every time someone asks something.

Index time (chunk, embed, store).

1. Chunk. Split your documents into passages: a paragraph, a section, a slice of a page. This step looks trivial and is not, for reasons we will come back to.
2. Embed. Turn each chunk into an embedding, a list of numbers that places similar meanings near each other. We took embeddings apart in their own explainer; here it is enough to know that closeness in that space means closeness in meaning, and that dense embedding retrieval beat older keyword search by 9 to 19 points on top-20 retrieval accuracy when it was introduced.
3. Store. Put each vector, with its chunk of text, into a vector database, which indexes embeddings for fast similarity search using approximate-nearest-neighbor methods like HNSW.

Query time (retrieve, augment, generate).

4. Retrieve. Embed the user’s question with the same model, then ask the database for the handful of chunks whose vectors are closest to it. Those are the passages most likely to hold the answer.
5. Augment. Drop those chunks into the context window alongside the original question. This is the step the name is built around, and it is where the two earlier explainers meet: embeddings chose the chunks, and the window is where they sit.
6. Generate. The model reads the question plus the retrieved chunks and writes an answer grounded in them, ideally citing which chunk each claim came from.

The shape to hold onto: the model is the last step, not the first. By the time it writes a word, the important decision, which passages it gets to see, has already been made by the retrieval half.

In RAG the model writes the answer, but it does not choose the evidence. Retrieval does that, before the model sees anything.

Why it works at all

The reason RAG took over is that it attacks the single biggest weakness of a bare language model: it answers from memory, and its memory is a blurry statistical average of its training data with no way to cite a source or know a fact has changed.

Retrieval fixes the worst of that. Because the model is reading actual passages from your data rather than recalling from its weights, the rate of invented answers drops. A 2021 study put it directly in its title: retrieval augmentation reduces hallucination in conversation. The original RAG work showed the same effect as a capability gain, setting state-of-the-art results on open-domain question answering by pairing a generator with a retriever over a Wikipedia index. And because the knowledge lives in the index rather than the weights, you can update it by re-indexing a document instead of retraining a model, which is why a survey of the field frames RAG as the practical way to keep a model current, domain-specific, and traceable to sources.

Where RAG breaks

This is the part most introductions skip, and it is the part that decides whether your system is trustworthy. RAG fails quietly, because a fluent wrong answer with a citation looks exactly like a fluent right one. The failures cluster in four places, and three of the four are upstream of the model entirely.

Bad chunking. How you split documents is the quiet foundation everything else stands on, and it is easy to get wrong. Split a clause from the condition that governs it and the retriever can only ever fetch half a rule, after which the model completes the other half plausibly and incorrectly. The research on RAG failure points names this directly: chunk boundaries and how chunks are consolidated are among the ways a system fails before the model is even involved.

Retrieval misses. The retriever can rank the wrong passage first, or miss the relevant one entirely, especially when thousands of documents look alike. The same failure-point work lists missing content and missed top-ranked documents as primary modes, and dense retrieval, for all that it beats keyword search, is only as good as the embedding model on your particular data. If retrieval hands over the wrong chunk, the model has no way to know it is answering the wrong question.

The model ignoring or contradicting the context. Even with the right passage in the window, the model does not always defer to it. A study of knowledge conflicts found a strong confirmation bias: models tend to stick with an answer that matches their training and discount retrieved evidence that contradicts it, though they will update when the external evidence is coherent and pointed. And if the right chunk lands in the middle of a long prompt, the lost-in-the-middle effect means the model may not really use it even though it is technically present. Whether the answer actually reflects the retrieved text is its own measurable property, which frameworks like RAGAS evaluate as faithfulness.

A stale index. The index is a photograph, not a live feed. If a document changes and the index is not rebuilt, retrieval keeps surfacing the old version, and the model answers confidently from a policy that was revised last quarter. Keeping the index synchronized with the source is ongoing maintenance, not a one-time setup, a point the survey literature treats as a core operational concern.

Three of the four ways RAG fails happen before the model runs. The model is usually the part working correctly on bad inputs.

The uncomfortable conclusion the failure-point research reaches is that you cannot validate a RAG system by inspection; you can only validate it in operation. A fluent, cited, wrong answer is indistinguishable from a right one until you check it against a known answer. So a serious RAG system ships with an evaluation set, a fixed list of questions with correct answers, run continuously, and it measures retrieval quality rather than trusting the model’s confidence.

Where retrieval goes next: agents and tools

So far retrieval is a fixed first step: every question triggers the same fetch. The next move is to let the model decide when to retrieve, what to search for, and whether one search was enough, looping until it has what it needs.

At that point retrieval stops being a pipeline stage and becomes a tool the model calls, which is exactly how agentic systems work: a model that plans, calls tools, and reacts to the results. Standardizing how a model reaches those tools and data sources is the job of the Model Context Protocol, which turns “retrieve from this index” into one more tool the model can invoke alongside querying a database or calling an API. RAG is the first and most common of those tools, and understanding it is what makes the agent versions legible.

What RAG actually buys you

RAG is not the model getting smarter. It is the model getting access. You are not changing what it knows in its weights; you are changing what it can see at the moment it answers, which is the same lever the context window gives you, now pointed at a whole library instead of a single prompt. That is why it became the default way to put a model on top of private or fast-moving data: it is cheaper than retraining and current in a way training never is.

The mistake is to treat the fetching as the hard part. The fetching is easy. The hard part is making sure the right passage was pulled, that the passage still says what you think it says, and that the model believed the passage over its own hunch. Get those three right and RAG is the most dependable pattern in applied AI. The library card was step one. Teaching the model to walk to the right shelf on its own, and to know when it has the wrong book, is the work.