# What is RAG, really?
> A from-scratch explainer of RAG: how chunk, embed, store, retrieve, augment, and generate fit together, why it reduces hallucination without ending it, and where it quietly breaks, in about 13 minutes.
- **Pillar:** Explainers
- **Author:** Aditya Marin Gasga (Founding Editor)
- **Published:** 2026-06-22T14:16:00.000Z
- **Tags:** rag, retrieval-augmented-generation, retrieval, vector-search, explainer
## TL;DR
RAG, retrieval-augmented generation, is a technique that lets a model answer from documents it was never trained on by fetching the most relevant passages at the moment of the question and adding them to the prompt. It runs in two halves: an indexing pass that chunks your documents, turns each chunk into an embedding, and stores it, and a query pass that retrieves the closest chunks, augments the prompt with them, and lets the model generate a grounded answer. It reduces hallucination because the model is reading real source text instead of recalling from memory, but it does not end it: bad chunking, retrieval misses, a model that overrides its sources, and a stale index all produce fluent answers that are wrong.
## Key takeaways
1. RAG has two halves. Indexing (chunk, embed, store) happens ahead of time; answering (retrieve, augment, generate) happens per question.
2. It leans entirely on embeddings for the retrieval step: the question and the chunks become vectors, and closeness in that space decides what the model reads.
3. RAG reduces hallucination because the model reads real passages instead of recalling from memory, but it does not eliminate it.
4. Most RAG failures are upstream of the model: bad chunk boundaries, a retriever that fetches the wrong passage, or an index that has gone stale.
5. A fluent answer with a citation can still be wrong, so a RAG system is only trustworthy with an evaluation set you run continuously.
import Figure from '~/components/article/Figure.astro';
import PullQuote from '~/components/article/PullQuote.astro';
import Callout from '~/components/article/Callout.astro';
RAG, retrieval-augmented generation, is the technique that lets a language model answer from documents it was never trained on. Instead of relying on what the model memorized, you fetch the most relevant passages from your own data at the moment of the question and put them in the prompt alongside it. The model still writes the answer; the retrieval step decides what it gets to read first.
That one idea, supplying knowledge at query time instead of baking it into the weights, was introduced as [retrieval-augmented generation in a 2020 paper](https://arxiv.org/abs/2005.11401) and is now the default way to put a model on top of private or fast-changing data. This piece builds on two earlier ones: the [embeddings explainer](/what-is-an-embedding-really), which covers how text becomes comparable vectors, and the [context window explainer](/what-is-a-context-window-really), which covers the space the retrieved text has to fit into.
## What RAG is doing, step by step
The whole pattern is six steps, and they split cleanly into two passes. The first pass happens once, ahead of any question. The second runs every time someone asks something.
**Index time (chunk, embed, store).**
1. **Chunk.** Split your documents into passages: a paragraph, a section, a slice of a page. This step looks trivial and is not, for reasons we will come back to.
2. **Embed.** Turn each chunk into an embedding, a list of numbers that places similar meanings near each other. We took embeddings apart in their [own explainer](/what-is-an-embedding-really); here it is enough to know that closeness in that space means closeness in meaning, and that [dense embedding retrieval beat older keyword search by 9 to 19 points on top-20 retrieval accuracy](https://arxiv.org/abs/2004.04906) when it was introduced.
3. **Store.** Put each vector, with its chunk of text, into a [vector database, which indexes embeddings for fast similarity search](https://www.pinecone.io/learn/vector-database/) using approximate-nearest-neighbor methods like [HNSW](https://arxiv.org/abs/1603.09320).
**Query time (retrieve, augment, generate).**
4. **Retrieve.** Embed the user's question with the same model, then ask the database for the handful of chunks whose vectors are closest to it. Those are the passages most likely to hold the answer.
5. **Augment.** Drop those chunks into the [context window](/what-is-a-context-window-really) alongside the original question. This is the step the name is built around, and it is where the two earlier explainers meet: embeddings chose the chunks, and the window is where they sit.
6. **Generate.** The model reads the question plus the retrieved chunks and writes an answer grounded in them, ideally citing which chunk each claim came from.
The shape to hold onto: the model is the last step, not the first. By the time it writes a word, the important decision, which passages it gets to see, has already been made by the retrieval half.
In RAG the model writes the answer, but it does not choose the evidence. Retrieval does that, before the model sees anything.
## Why it works at all
The reason RAG took over is that it attacks the single biggest weakness of a bare language model: it answers from memory, and its memory is a blurry statistical average of its training data with no way to cite a source or know a fact has changed.
Retrieval fixes the worst of that. Because the model is reading actual passages from your data rather than recalling from its weights, the rate of invented answers drops. A 2021 study put it directly in its title: [retrieval augmentation reduces hallucination in conversation](https://arxiv.org/abs/2104.07567). The original RAG work showed the same effect as a capability gain, [setting state-of-the-art results on open-domain question answering](https://arxiv.org/abs/2005.11401) by pairing a generator with a retriever over a Wikipedia index. And because the knowledge lives in the index rather than the weights, you can update it by re-indexing a document instead of retraining a model, which is why a survey of the field frames RAG as the practical way to keep a model [current, domain-specific, and traceable to sources](https://arxiv.org/abs/2312.10997).
RAG **reduces** hallucination; it does not end it. The model reads real text instead of guessing from memory, which lowers the rate of invented answers. It does nothing to guarantee the retrieved text was the right text, or that the model read it correctly.
## Where RAG breaks
This is the part most introductions skip, and it is the part that decides whether your system is trustworthy. RAG fails quietly, because a fluent wrong answer with a citation looks exactly like a fluent right one. The failures cluster in four places, and three of the four are upstream of the model entirely.
**Bad chunking.** How you split documents is the quiet foundation everything else stands on, and it is easy to get wrong. Split a clause from the condition that governs it and the retriever can only ever fetch half a rule, after which the model completes the other half plausibly and incorrectly. The [research on RAG failure points names this directly](https://arxiv.org/abs/2401.05856): chunk boundaries and how chunks are consolidated are among the ways a system fails before the model is even involved.
**Retrieval misses.** The retriever can rank the wrong passage first, or miss the relevant one entirely, especially when thousands of documents look alike. The same failure-point work lists missing content and missed top-ranked documents as primary modes, and dense retrieval, for all that it [beats keyword search](https://arxiv.org/abs/2004.04906), is only as good as the embedding model on your particular data. If retrieval hands over the wrong chunk, the model has no way to know it is answering the wrong question.
**The model ignoring or contradicting the context.** Even with the right passage in the window, the model does not always defer to it. A study of knowledge conflicts found a strong [confirmation bias: models tend to stick with an answer that matches their training and discount retrieved evidence that contradicts it](https://arxiv.org/abs/2305.13300), though they will update when the external evidence is coherent and pointed. And if the right chunk lands in the middle of a long prompt, the [lost-in-the-middle effect](https://arxiv.org/abs/2307.03172) means the model may not really use it even though it is technically present. Whether the answer actually reflects the retrieved text is its own measurable property, which frameworks like [RAGAS evaluate as faithfulness](https://arxiv.org/abs/2309.15217).
**A stale index.** The index is a photograph, not a live feed. If a document changes and the index is not rebuilt, retrieval keeps surfacing the old version, and the model answers confidently from a policy that was revised last quarter. Keeping the index synchronized with the source is ongoing maintenance, not a one-time setup, a point the [survey literature treats as a core operational concern](https://arxiv.org/abs/2312.10997).
Three of the four ways RAG fails happen before the model runs. The model is usually the part working correctly on bad inputs.
The uncomfortable conclusion the failure-point research reaches is that you [cannot validate a RAG system by inspection; you can only validate it in operation](https://arxiv.org/abs/2401.05856). A fluent, cited, wrong answer is indistinguishable from a right one until you check it against a known answer. So a serious RAG system ships with an evaluation set, a fixed list of questions with correct answers, run continuously, and it measures retrieval quality rather than trusting the model's confidence.
## Where retrieval goes next: agents and tools
So far retrieval is a fixed first step: every question triggers the same fetch. The next move is to let the model decide when to retrieve, what to search for, and whether one search was enough, looping until it has what it needs.
At that point retrieval stops being a pipeline stage and becomes a tool the model calls, which is exactly how [agentic systems](/what-is-agentic-ai) work: a model that plans, calls tools, and reacts to the results. Standardizing how a model reaches those tools and data sources is the job of the [Model Context Protocol](/what-is-mcp-protocol), which turns "retrieve from this index" into one more tool the model can invoke alongside querying a database or calling an API. RAG is the first and most common of those tools, and understanding it is what makes the agent versions legible.
## What RAG actually buys you
RAG is not the model getting smarter. It is the model getting access. You are not changing what it knows in its weights; you are changing what it can see at the moment it answers, which is the same lever the [context window](/what-is-a-context-window-really) gives you, now pointed at a whole library instead of a single prompt. That is why it became the default way to put a model on top of private or fast-moving data: it is cheaper than retraining and current in a way training never is.
The mistake is to treat the fetching as the hard part. The fetching is easy. The hard part is making sure the right passage was pulled, that the passage still says what you think it says, and that the model believed the passage over its own hunch. Get those three right and RAG is the most dependable pattern in applied AI. The library card was step one. Teaching the model to walk to the right shelf on its own, and to know when it has the wrong book, is the work.
## FAQ
### What does RAG stand for?
Retrieval-augmented generation. 'Retrieval' is the step that fetches relevant text from your own data, 'augmented' means that text is added to the prompt, and 'generation' is the model writing the answer from it. The name is literally the pipeline in order.
### How is RAG different from fine-tuning a model?
Fine-tuning changes the model's weights so new knowledge is baked in, which is expensive and goes stale the moment your data changes. RAG leaves the weights alone and supplies knowledge at the moment of the question by retrieving it. Fine-tuning teaches a style or skill; RAG supplies current facts. Many production systems use a base model with RAG and skip fine-tuning entirely.
### Does RAG stop a model from hallucinating?
It reduces hallucination, because the model is answering from retrieved source text rather than from memory, but it does not eliminate it. The model can still misread a passage, blend it with its own assumptions, or contradict it outright. Grounding lowers the rate of invented answers; it does not guarantee a correct one.
### Why does RAG give wrong answers even when the right document exists?
Usually the failure is before the model. The passage may have been split badly during chunking, the retriever may have ranked the wrong chunk first, or the right chunk may have landed in the middle of a long context where recall is weakest. The model can only work with what retrieval handed it.
### Do I still need a large context window if I use RAG?
RAG reduces how much you need to stuff into the window, because you send a handful of relevant chunks instead of a whole corpus. The two are complementary: retrieval decides what is worth including, and the context window is the space it gets included in. See the companion piece on the context window for that side of it.