# What is a context window, really?
> A from-scratch explainer of the context window: the shared token budget a model reads at once, why long context costs more and recalls less, and when retrieval beats a bigger window, in about 12 minutes.
- **Pillar:** Explainers
- **Author:** Aditya Marin Gasga (Founding Editor)
- **Published:** 2026-06-22T14:14:00.000Z
- **Tags:** context-window, long-context, tokens, llm, explainer
## TL;DR
A context window is the fixed amount of text, measured in tokens, that a model can read in a single request, and it is shared by everything at once: the system prompt, the entire conversation so far, any documents you add, and the answer the model is about to write. When the total exceeds the window, the oldest text falls out of view and the model can no longer use it, which is why long chats lose their beginnings. Bigger windows help, but they cost more, run slower, and recall less from the middle, so the real skill is deciding what earns a place in the budget rather than trying to fit everything in.
## Key takeaways
1. A context window is a token budget, not a memory. The system prompt, the conversation, retrieved documents, and the answer itself all draw from one fixed pool.
2. When a conversation outgrows the window, the earliest tokens fall out of view. The model has not forgotten them; it can no longer see them.
3. Bigger context is not free. You pay per input token, latency climbs with input length, and recall sags for facts buried in the middle (the lost-in-the-middle effect).
4. Effective context is usually shorter than the number on the box. Models use the start and end of a long input more reliably than the middle.
5. When your data is larger than any window, or changes often, the fix is retrieval, not a bigger window.
import Figure from '~/components/article/Figure.astro';
import PullQuote from '~/components/article/PullQuote.astro';
import Callout from '~/components/article/Callout.astro';
A context window is the amount of text a model can read at one time, measured in tokens, and everything a model does in a single request has to fit inside it. The system prompt, the whole conversation so far, any documents you pasted, and the answer the model is about to write all draw from that one budget. Once you picture it as a budget rather than a memory, most of the confusing behavior of chatbots starts to make sense.
This piece sits one step on from our [explainer on LLMs](/what-is-an-llm-really), which covered tokens and the loop that predicts them. Here we look at the space those tokens live in.
## What a context window actually is
Anthropic's own documentation puts it plainly: the context window is [all the text a model can reference when generating a response, including the response itself](https://platform.claude.com/docs/en/docs/build-with-claude/context-windows). That last clause is the part people miss. The output is not free; it is drawn from the same pool as the input. If you hand the model 195,000 tokens of documents inside a 200,000-token window, you have left it room for a 5,000-token answer and no more.
So the window holds four kinds of thing, and they compete:
- The **system prompt**: the standing instructions, the persona, the rules, plus any tool definitions.
- The **conversation so far**: every prior message from you and the model, kept verbatim.
- **Retrieved or pasted material**: the PDF, the code, the search results, the database rows.
- The **answer**: the tokens the model is generating right now.
The same documentation describes it as the model's working memory for a single request, distinct from the vast training data baked into its weights. Training is what the model knows in general; the context window is what it can see right now. They are different stores, and only one of them resets every time you press enter.
## Why a long chat loses its beginning
Run a chat long enough and the model starts acting as though the first thing you told it never happened. This is not the model being forgetful in a human way. It is arithmetic.
Most chat interfaces manage the window on a first-in, first-out basis: as the running total of tokens climbs past the limit, [the earliest messages drop out of the window](https://platform.claude.com/docs/en/docs/build-with-claude/context-windows) to make room for new ones. The text that scrolled off is no longer part of what the model sees on this turn. It cannot reference what is not in front of it, so the careful instruction you gave in message one quietly stops applying around message forty.
The model has not forgotten the start of the chat. It can no longer see it. Those are different problems, and only one of them is fixable from your side.
That distinction matters because the fix is in your hands. If something must stay true for the whole session, it belongs in the system prompt (which usually stays pinned) rather than in an early chat message (which scrolls away). Knowing the window is FIFO turns a mysterious behavior into a layout decision.
## Bigger is not free: cost, latency, and recall
The obvious move, once you understand the budget, is to want a much bigger one. Sometimes that is right. But three real costs grow with the size of what you actually send.
**Money.** API usage is billed per token, and input tokens are most of the bill on a long prompt. The price is not even flat across the whole range: when a request to Claude Sonnet 4 runs past 200,000 tokens, [the input rate rises from $3 to $6 per million tokens, and output from $15 to $22.50](https://claude.com/blog/1m-context). So a giant prompt can cost more than twice as much per token as a small one, on top of simply having more tokens to pay for.
**Latency.** Before a model writes its first word, it has to read everything you sent, a phase called prefill. The transformer's attention mechanism, introduced in the 2017 paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762), compares every token with every other token, so the work grows faster than linearly as the input lengthens. The memory needed to hold all those intermediate values, the KV cache, [grows with sequence length too, and the cost shows up most on long sequences](https://arxiv.org/abs/2309.06180). In practice that means a very long prompt adds a noticeable wait before the answer starts.
**Recall.** This is the least intuitive cost and the most important. Models do not read a long context evenly. The 2023 study [Lost in the Middle](https://arxiv.org/abs/2307.03172) found that performance is highest when the relevant information sits at the very beginning or the very end of the input, and [degrades significantly when the model has to use something buried in the middle](https://arxiv.org/abs/2307.03172), even for models built for long context. Worse, the advertised window is not the usable window: the [RULER benchmark](https://arxiv.org/abs/2404.06654) found that of models claiming 32,000 tokens or more, only about half held up at 32,000, and almost all degraded as the input grew. Anthropic's documentation gives this a name, [context rot](https://platform.claude.com/docs/en/docs/build-with-claude/context-windows): as the token count climbs, accuracy and recall fall.
Every token you add is taxed three ways: in **money** (you pay per input token, and the rate can rise past a threshold), in **latency** (the model must read it all before replying), and in **recall** (facts in the middle of a long input are the hardest for the model to use).
## How big are the windows now
Five years ago a 2,000-token window was normal. Today the frontier sits between 200,000 tokens and 1 million. Several Claude models, including Opus 4.8 and Sonnet 4.6, [carry a 1 million-token window, while others hold 200,000](https://platform.claude.com/docs/en/docs/build-with-claude/context-windows). Google states that [Gemini accepts 1 million tokens or more](https://ai.google.dev/gemini-api/docs/long-context). A million tokens is roughly 750,000 words, comfortably more than a long novel.
That growth is real and useful. It is also where the trap lives. The number on the box is a capacity, not a recommendation. Filling it is exactly the behavior that triggers the cost, latency, and recall taxes above. The right question is never how much fits; it is how little you can get away with sending.
## How to fit what actually matters
Once you treat the window as a budget, a handful of moves follow directly from how it works.
**Put the load-bearing parts at the edges.** Because the middle is the weak spot, the instructions and facts the answer most depends on should sit near the top or the bottom of the prompt, not stranded in the center of a long block. This is a direct, free application of the lost-in-the-middle finding.
**Keep the stable parts first, so caching can pay you back.** If the opening of your prompt does not change between requests (a fixed system prompt, a reference document), prompt caching lets the model reuse that work. Anthropic's cache reads are [billed at one-tenth of the normal input price and improve time-to-first-token for long documents](https://platform.claude.com/docs/en/docs/build-with-claude/prompt-caching). Order your prompt stable-first and you turn a repeated cost into a near-free one.
**Send the smallest high-signal set, not everything you have.** Anthropic's own guidance for builders is to treat [context as a finite resource with diminishing returns and to find the smallest set of high-signal tokens](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) that gets the job done. More context is not more help once the signal-to-noise ratio drops.
The question is never how much fits in the window. It is how little you can send and still get the answer right.
## When the fix is retrieval, not a bigger window
There is a point where no window is big enough, and reaching for a larger one is the wrong instinct. If your knowledge base is a hundred thousand support tickets, or a wiki that changes daily, you cannot paste it, and you would not want to: even if it fit, you would be paying to read all of it and asking the model to find the needle in the middle, which is precisely where it is weakest.
The alternative is to fetch only the few passages that bear on the question and send those. That pattern, retrieving relevant text at the moment of the question and adding it to the prompt, was introduced as [retrieval-augmented generation in 2020](https://arxiv.org/abs/2005.11401) and it is now the default architecture for answering from private or changing data. A big window is the right tool when you genuinely need the whole thing in view at once: one long contract, a single codebase under analysis. Retrieval is the right tool when you need the right 5,000 tokens out of millions.
## The budget you are really managing
The mental shift is small and it changes everything downstream. A context window is not a bucket you fill; it is a budget you allocate. The system prompt, the history, the documents, and the answer are all bidding for the same fixed space, the oldest items get evicted first, and the middle is the worst seat in the house. Manage it as a budget and the model's odd behavior turns predictable: it loses the start of long chats because that text aged out, it slows and costs more on huge prompts because it must read all of them, and it misses facts you definitely included because they were buried where recall is thinnest.
When what you need the model to read is bigger than any window, the answer is not a bigger window. It is retrieval, and that has its own mechanism worth understanding on its own terms. We take it apart next, in [what RAG really is](/what-is-rag-really).
## FAQ
### What is a context window in simple terms?
It is the amount of text a language model can read at one time, counted in tokens (a token is roughly three-quarters of a word). Everything the model needs for a single reply has to fit inside it: the instructions, the conversation so far, any files you paste, and the reply itself. Think of it as the model's working memory for one request, not a long-term store.
### How big are context windows in 2026?
Frontier models run from about 200,000 tokens to 1 million. Several Claude models and Google's Gemini both reach 1 million tokens, enough for a long book or a mid-sized codebase. A few research systems advertise more. But the usable amount is normally smaller than the advertised number, because recall degrades well before the hard limit.
### Why does the model forget what I said at the start of a long chat?
It does not forget so much as lose sight of it. Most chat interfaces run the window on a first-in, first-out basis: once the running total of tokens passes the limit, the earliest messages drop out of the window and the model no longer sees them. The text is gone from its view, so it cannot use it.
### Does a bigger context window always give better answers?
No. A larger window lets you include more, but every extra token costs money and adds latency, and accuracy tends to fall as the window fills, a pattern vendors now call context rot. Curating the right few thousand tokens usually beats dumping in hundreds of thousands and hoping the model finds the part that matters.
### What is the difference between a context window and model memory?
The context window is per request: it resets every time and only holds what you send. Product memory features fake continuity by saving notes about you and re-injecting them into the prompt on the next request, which means those notes also consume the same window. The underlying model still starts each request with only what fits in the budget.