# How RAG fails in production, and how to fix it
> The intermediate companion to our RAG explainer: the four ways RAG degrades in production (chunking, retrieval, faithfulness, a stale index), the fixes that hold, and the evaluation that makes them visible.
- **Pillar:** Explainers
- **Author:** Aditya Marin Gasga (Founding Editor)
- **Published:** 2026-06-22T15:20:00.000Z
- **Tags:** rag, retrieval, rag-evaluation, production, explainer
## TL;DR
A RAG demo built in an afternoon will quietly degrade in production, and the cause is almost never the model. Four parts fail: chunking that splits meaning, retrieval that misses or drifts as the corpus grows, a model that ignores or contradicts the passages it was handed, and an index that goes stale. Each has a concrete fix (structure-aware chunking, hybrid search and reranking, tighter grounding, and a freshness pipeline), but none of them are visible without an evaluation set that measures retrieval and faithfulness on every change.
## Key takeaways
1. A demo and a production RAG system fail differently. The demo hides failures because nothing is measuring them.
2. Chunking sets the ceiling: if a passage is split badly at index time, no retriever or model can recover it. Split on structure, not a fixed token count.
3. Dense retrieval alone misses exact terms and rare identifiers. Hybrid search plus a reranker fixes most real-world retrieval misses.
4. Even with the right passage retrieved, the model can ignore or contradict it. Send fewer, better-ordered chunks and instruct it to ground and cite.
5. You cannot operate RAG without an evaluation set. Retrieval metrics and faithfulness scoring, run continuously, are the only way the failures become visible.
import Figure from '~/components/article/Figure.astro';
import PullQuote from '~/components/article/PullQuote.astro';
import Callout from '~/components/article/Callout.astro';
A retrieval-augmented generation demo is easy to build and easy to trust, which is the problem. The version you stand up in an afternoon answers your test questions correctly, so you ship it, and then it degrades in production in ways that are hard to see and almost never the model's fault. This piece is about that gap: the four places RAG breaks once real documents and real traffic arrive, and the specific fix for each.
It assumes you already know the mechanism. If chunk, embed, retrieve, augment, and generate are not yet familiar, start with [what RAG really is](/what-is-rag-really); here we skip the assembly and go straight to where the assembled thing fails.
## Why the demo hides the failures
A demo asks a handful of questions whose answers sit in obvious places, against a corpus small enough that retrieval can hardly miss. Production asks questions you never tried, against a corpus that grows and changes under you. The failures were there on day one; you were just not measuring them, so the demo looked clean.
There are four of them, and they live at different stages of the pipeline. Three happen before the model ever runs. The fourth is the model misusing context that was handed to it correctly. Each has a known fix, and the diagram below is the whole article in one frame.
Three of the four failures (chunking, retrieval, freshness) happen **before the model runs**. The fourth is the model misusing context it was handed correctly. None of them are visible without measurement, which is why evaluation is the fix that makes the other fixes possible.
## Chunking decides the ceiling
How you split documents at index time sets a hard limit on everything downstream, and a fixed-size splitter is the usual culprit. Cut every 500 tokens and you will eventually slice a clause from the condition that governs it, a figure from its unit, or a table row from its header. Once a passage is split wrong, no retriever can fetch the missing half and no model can reconstruct it. The [research cataloguing RAG failure points flags chunking and chunk consolidation as failures that occur before the model is ever involved](https://arxiv.org/abs/2401.05856).
The fix is to stop splitting on a token count and start splitting on structure: headings, paragraphs, list items, table boundaries. Add a little overlap so a boundary is a seam rather than a hard cut, and attach metadata (source, section, date) so retrieval can filter. When the best-retrieving unit is small but the model needs more around it, retrieve the small chunk and feed its parent section. For corpora where answers span many passages, hierarchical indexing helps: [RAPTOR builds a tree of recursive summaries and shows significant improvements over flat-chunk retrieval](https://arxiv.org/abs/2401.18059), because a summary node can answer a broad question that no single leaf chunk covers.
## Retrieval has to work on the whole corpus, not the demo
Dense embeddings capture meaning, which is exactly why they miss exact terms. A query for error code E-4021 or a specific part number can rank a semantically similar passage above the literal match, because the embedding cares about what the text means, not the characters it contains. [Dense retrieval beats keyword search by 9 to 19 points on top-20 accuracy on average](https://arxiv.org/abs/2004.04906), and the average is real, but it hides the queries where a keyword match is the only right answer. Then the corpus grows, near-duplicates multiply, and the passage you need slips out of the top results.
The fix is three moves, in order. Run hybrid search: combine the dense similarity score with a sparse keyword signal like BM25, so meaning and exact terms both count. Then rerank: a cross-encoder re-scores the top candidates against the query and reorders them, and [BERT-based reranking improved passage ranking by 27 percent relative MRR on MS MARCO](https://arxiv.org/abs/1901.04085). And rewrite the query before you search: [generating a hypothetical answer and embedding that (the HyDE method) measurably improves zero-shot retrieval](https://arxiv.org/abs/2212.10496), because a hypothetical answer sits closer in embedding space to the real passage than the bare question does. Embeddings still do the heavy lifting (the [embeddings explainer](/what-is-an-embedding-really) covers why); in production they are the first stage, not the only one.
## When the model fights the context
Retrieval can hand over the right passage and the model can still get it wrong. Models carry a [confirmation bias toward their own training: they discount retrieved evidence that contradicts what they already believe](https://arxiv.org/abs/2305.13300), unless that evidence is coherent and pointed enough to override it. And the way you pack the prompt makes it worse. Stuff in twenty chunks and the decisive one may land in the middle, where the [lost-in-the-middle effect](https://arxiv.org/abs/2307.03172) shows models read least reliably; performance is highest at the start and end of the context and sags in between. Reaching for a bigger window is not the escape it looks like, because [usable context is shorter than the advertised length and accuracy drops as you fill it](https://arxiv.org/abs/2404.06654).
So the fix runs against the instinct to send more. Send fewer, stronger chunks, and use the reranker to place the best evidence at the start or end rather than the middle. Instruct the model to answer only from the supplied passages, to cite the chunk behind each claim, and to say it does not know when the passages do not support an answer. Then check that it actually did: [faithfulness, whether the answer is grounded in the retrieved text, is a measurable property that frameworks like RAGAS score without needing a reference answer](https://arxiv.org/abs/2309.15217).
Retrieval can hand the model the right passage and the model can still answer from its own memory. Grounding is something you instruct and then measure, never something you assume.
## The index ages out
The index is a snapshot taken at ingest time, and the world it describes keeps moving. A document gets revised and the old chunk keeps surfacing, so the model answers from a policy that changed last quarter. A document is deleted and its chunks linger. New documents arrive and stay invisible until something indexes them. The [survey of the field treats outdated knowledge and continuous updates as a core operational concern](https://arxiv.org/abs/2312.10997), not a setup step you do once.
The fix is to treat indexing as a running pipeline rather than a one-time load. Re-embed on change, ideally triggered by the source system's own update events, version chunks so you can tell which revision produced an answer, expire stale entries, and monitor the lag between when a source changes and when the index reflects it. A freshness check is far cheaper than a confident, well-cited answer about a policy that no longer exists.
## You cannot fix what you do not measure
The reason all four failures stay hidden is that a fluent, cited, wrong answer looks exactly like a fluent, cited, right one. The failure-point research reaches an uncomfortable conclusion on this: a [RAG system can only be validated in operation](https://arxiv.org/abs/2401.05856), not reasoned about in advance. Which means the real deliverable is not the pipeline. It is the evaluation set.
Build a fixed set of questions with known-good answers drawn from your actual corpus, and measure the two halves separately. Retrieval: did the correct passage make the top results, scored by recall and rank. Generation: is the answer faithful to what was retrieved, and does it answer the question that was asked. [Reference-free frameworks like RAGAS score faithfulness and relevance without a hand-labeled answer for every case](https://arxiv.org/abs/2309.15217), so you can run the whole suite on every index change, every prompt edit, and every model swap, in CI, the way you run tests. Skip it and every fix above is a guess you cannot check.
## Where the fixes lead: agentic retrieval
Everything so far treats retrieval as one fixed pass: the same fetch fires for every question. The next step is to let the model run the loop, deciding whether it needs to retrieve at all, judging whether what came back is sufficient, rewriting the query, and searching again before it commits to an answer. [Self-RAG trains a model to retrieve on demand and critique its own output, and reports gains in factuality and citation accuracy](https://arxiv.org/abs/2310.11511) over a fixed retrieve-then-read setup. At that point retrieval is a tool the model chooses to call, which is the bridge to [agents](/what-is-agentic-ai) and to the [Model Context Protocol](/what-is-mcp-protocol) that standardizes how a model reaches tools and data in the first place.
## What you are actually shipping
A RAG pipeline is a few hundred lines of glue, and almost all of it is replaceable. You will change the embedding model, swap the chunker, add a reranker, move vector stores. The one part that does not change, the part that tells you whether any of those swaps helped or hurt, is the evaluation set. That is the asset. Teams that treat RAG as a one-time integration ship the demo and inherit the slow decay that follows. Teams that treat it as a measured system build the evaluation set first and let it catch the regressions the demo would have waved through. The pipeline produces the answer. The evaluation set is the only thing that tells you the answer was right.
## FAQ
### Why does my RAG system get worse over time?
Two reasons, both operational. The index ages: documents get revised, deleted, or added, and a snapshot taken at ingest keeps serving the old version until you re-index. And the corpus grows: more near-duplicate passages means the right one is less likely to land in the top results for a given query. Neither is the model degrading; the parts around it are drifting, and only an evaluation set will show you when.
### What is hybrid search in RAG?
Running two retrievers and combining their results: a dense (embedding) search that matches meaning, and a sparse keyword search like BM25 that matches exact terms. Dense search alone blurs identifiers, codes, and rare jargon; keyword search alone misses paraphrases. Combining them, then reranking the merged list, catches both and is the single highest-impact retrieval fix in most production systems.
### What does a reranker do, and do I need one?
A reranker is a second-stage model (usually a cross-encoder) that re-scores the top candidates from your first retrieval against the query and reorders them, so the most relevant passage lands first. First-stage retrieval optimizes for speed over a whole corpus; the reranker optimizes for precision over a short list. If your retrieval recall is decent but the right passage is not at the top, a reranker is usually the fix.
### How do I evaluate a RAG system?
Build a fixed set of questions with known-good answers from your real corpus, and measure the two halves separately. Retrieval: did the correct passage make the top results (recall and rank). Generation: is the answer faithful to what was retrieved, and does it actually answer the question. Run it on every index change and prompt edit, the way you run tests, so a regression is caught before users see it.
### Why does the model ignore the documents I gave it?
Two common causes. The model has a confirmation bias toward its own training and may discount a retrieved passage that contradicts it unless that passage is clear and pointed. And if you send many chunks, the relevant one can land in the middle of the prompt, where models read least reliably. The fixes are to send fewer, better-ordered chunks and to instruct the model to answer only from the passages and cite them.