What a context window is, and why long AI chats start losing the thread

A context window is the pool of tokens a model can hold inside a single request — everything you paste in, everything it has already said back, and the answer it is still writing. OpenAI's API documentation defines it as the maximum number of tokens usable in one request, counting input, output and reasoning tokens together. When that pool fills, something has to give.

Every load-bearing number below comes from vendor documentation read on August 20, 2026: OpenAI's conversation-state guide and its token help page, Anthropic's context-windows page for Claude, and Google's Gemini long-context guide. The one non-vendor result is “Lost in the Middle,” by Nelson F. Liu and colleagues at Stanford, UC Berkeley and Samaya AI, published in Transactions of the Association for Computational Linguistics. Model versions are named wherever a figure attaches to one. Nobody here ran a benchmark; these are the published figures, and they change month to month.

What is actually inside the window?

More than your last message. Anthropic's documentation describes the context window as everything the model can reference while generating a response, including the response itself — its working memory, as distinct from its training data. In practice that bucket holds several things you never typed.

Per that same Anthropic page, what counts toward the window includes the system prompt, every message in the conversation so far, tool results, images and attached documents, tool definitions, and the model's own generated output. On the models that keep them, extended thinking blocks stay in context and get billed as input tokens on the next turn. Other models — earlier Opus and Sonnet releases, and all Haiku models — have those thinking blocks stripped automatically to preserve capacity.

The mechanic that catches people out is accumulation. Anthropic's documentation spells out the loop: each turn sends the entire prior conversation plus your new message as input, then the model's reply becomes input for the turn after that. A chat does not stay the size it looks. It grows every time either party speaks.

How many words is a million tokens?

Roughly 750,000 English words, using OpenAI's own rule of thumb. Its help documentation puts one token at about three-quarters of a word, or 100 tokens to 75 words. Tokens are not words, though — OpenAI describes them as building blocks that can run from a single character to a full word depending on language and context.

Google's Gemini long-context guide translates the same ceiling into things you can picture. One million tokens, per that documentation, is about 50,000 lines of code at the standard 80 characters per line, or eight average-length English novels, or transcripts of more than 200 average-length podcast episodes. Also, per Google, every text message you have sent in the last five years.

Which is the honest reason these numbers feel abstract. Nobody arrives at their desk with eight novels to process. They arrive with a 40-page contract, a quarter of Slack history and a meeting transcript, and the question is whether all three fit at once.

How big are the windows, as documented?

Here is what the vendors publish, as of August 20, 2026. Treat every row as a snapshot: these figures move, and the model names attached to them move faster.

Model (as named in vendor docs)	Documented context window	Documented max output
gpt-4o-2024-08-06	128k tokens total	16,384 tokens
Claude Opus 5, Sonnet 5	1M tokens	up to 128k tokens
Claude Sonnet 4.5, Haiku 4.5	200k tokens	not stated on this page
Gemini (long-context guide)	1M tokens	not stated on this page

Note the second column exists at all. On the OpenAI side the window is a single shared budget — input, output and reasoning tokens drawing on one pot — so a long paste quietly shrinks the room left for the answer. Anthropic documents a separate output ceiling of up to 128k tokens for its million-token models. Same idea, different accounting.

If the window is huge, why does quality still slip?

Because capacity and attention are not the same thing. The Liu et al. paper found a U-shaped performance curve: models did best when the relevant information sat at the very start or the very end of a long input, and measurably worse when it sat in the middle.

The numbers are specific. In the paper's 20-document multi-document question-answering setting, GPT-3.5-Turbo scored roughly 75.8% when the answer-bearing document was first, about 53.8% when it was in the middle, and about 63.2% when it was last. The closed-book baseline — the model answering with no documents at all — was 56.1%. At 20 to 30 documents, middle-position performance fell below that baseline, meaning the extra material actively hurt.

The authors also report that extended-context versions of models showed nearly identical curves to their standard counterparts on smaller inputs. A bigger window, on this evidence, does not automatically mean better use of it.

Google's documentation concedes a version of the same limit from the vendor side: single-needle retrieval is described as roughly 99% accurate, but performance can vary widely by context, and hunting several pieces of information at once may need separate API calls to hold that standard.

What happens when you hit the ceiling?

It depends whose ceiling. OpenAI's guide warns that an oversized prompt risks exceeding the model's allocated context window, which may result in truncated output, and that tokens generated beyond the limit may be truncated in API responses. Truncation, not an error — the reply just stops.

Anthropic documents a stricter and more legible failure. Input over the window returns a 400 invalid_request_error reading “prompt is too long.” On Claude 4.5 and later, a request whose input plus requested output exceeds the window is accepted anyway, and generation halts with a stop reason of model_context_window_exceeded. The page also describes server-side compaction, which summarizes earlier parts of a long conversation automatically, and context editing, which clears out old tool results and thinking blocks.

Compaction is worth sitting with. It means a long chat can keep running past its nominal limit while quietly replacing your earlier turns with a summary of them. Nothing errors. The detail is simply no longer there.

What to do with this at your desk

1. Put the thing that matters first or last. The U-shaped curve in the Liu paper is the single most actionable finding here: buried instructions are the ones that get missed.
2. Start a fresh chat for a genuinely new task rather than continuing an old one. Per Anthropic's documentation, every prior turn is re-sent as input on every turn; a clean thread is a cheaper and less crowded one.
3. Ask one retrieval question at a time when the material is long, which is what Google's guide recommends for multiple-needle work.
4. Estimate before you paste. Both OpenAI and Anthropic document tooling for counting tokens ahead of a request — a tokenizer and a token-counting API respectively.
5. Reuse a large fixed document via caching where your tooling supports it. Google's documentation puts context caching at roughly a 4x cost reduction versus standard input and output pricing for repeated queries over the same cached context.

What these sources do not establish

They do not establish which model handles long inputs best today. The Liu paper's figures come from an older generation of models and were never intended as a current leaderboard; the vendor pages document their own products and do not compare against rivals. Nothing here tests whether compaction preserves the details you care about, because no vendor page quantifies that. And none of these sources says what a million-token window costs you in practice — pricing lives on separate pages and changes on its own schedule.

If you use AI occasionally, the whole of this article collapses to one habit: put the important sentence at the top or the bottom, and start a new chat more often than feels necessary. If it is your daily driver, the numbers in the table are worth re-checking every few months, because the only reliable thing about them is that they will be wrong by winter.

For a related business news perspective, read Is AI actually taking entry-level jobs? Here's what the data says.