Your context window is smaller than you think

Every few months a lab ships a bigger context window and a portion of the industry concludes that context engineering is over. Just put everything in the prompt. The retrieval stack was scaffolding, the summarisation logic was a hack, and now we can delete both.

It is a reasonable inference and it is wrong, in a specific and measurable way. The window a model advertises is the number of tokens it will accept without erroring. It is not the number of tokens across which it holds accuracy. Those are different quantities, they can differ by a factor of eight, and for anyone building long-running agents the gap between them is where most production failures live.

This post covers what the evidence shows, the families of algorithms people use in response, and a decision framework for choosing between them.

Advertised is not effective

The cleanest measurement here is RULER, from NVIDIA. It probes models with thirteen synthetic tasks spanning retrieval, multi-hop variable tracing, aggregation and question answering, then asks a blunt question: at what length does quality actually fall below a usable threshold?

Across seventeen models that all claimed windows of 32,000 tokens or more, only four held satisfactory performance at 32K. Not at 128K. At 32K.

Held quality at 32K4 / 17

Fell below threshold13 / 17

That last clause matters more than the headline. The models pass the easy test and fail the hard one, which means the benchmark most people cite to justify trusting long context is precisely the benchmark that cannot detect the problem.

Why the easy test lies

Needle-in-a-haystack works by hiding a sentence in a pile of text and asking the model to find it. The catch is that the question usually shares vocabulary with the needle. The model can succeed on lexical overlap alone, which is not the skill you need in production, where a user asks about a concept and the relevant passage never uses their words.

NoLiMa, from Adobe Research, removes the overlap. Same structure, but the question and the needle share no literal terms, so answering requires an actual associative hop. Roughly seventy-two percent of its pairs need world knowledge to connect. The results are stark.

Short-context baseline99.3%

At 32K, no lexical overlap69.7%

There is a second, separate effect layered on top of this. Liu et al. established that position matters independently of length: accuracy traces a U-curve, strong when the evidence sits at the start or end of the context and materially weaker in the middle. Chroma's context rot study, covering eighteen frontier models, then showed that length degrades performance even when position is held favourable and the task is as simple as copying text. Two distinct failure modes, frequently conflated.

Chroma also found something useful for anyone choosing a model. Claude models decayed the slowest across the set, and they tended to abstain when uncertain rather than confidently return a distractor. Whether you prefer a wrong answer or no answer depends on your application, but it is a real behavioural difference and worth knowing about.

Agents do not fail because one document was too long. They fail because four hundred tool results were individually reasonable.

Agents break differently

Everything above concerns single-shot tasks: one large prefill, one answer. Agents fail through a different mechanism.

An agent runs a loop. It gathers context, acts, verifies, repeats. Every iteration appends tool output, error text and reasoning to a history that only grows. Nobody makes a decision to blow the budget. It accumulates, one plausible increment at a time, and each increment adds distractors that compound the rot problem described above.

This is why Claude Code caps tool responses at 25,000 tokens by default. It is not a capability limit, it is an accumulation defence.

The taxonomy

There are roughly nine families of technique in the literature. The most useful way to organise them is not by mechanism but by layer, because the layer tells you whether you can actually reach the technique from where you sit.

Application layer

Compaction summarises the conversation and reinitialises with the summary. Claude Code does this by passing the message history back through the model to preserve architectural decisions, unresolved bugs and implementation details while discarding redundant tool output, then resuming with the compressed summary plus the five most recently touched files. When tuning the compaction prompt, maximise recall first and worry about precision second. Dropping something that turns out to matter three turns later is the expensive failure.

Retrieval splits into two philosophies. Classic RAG pre-indexes everything into vectors. Agentic retrieval keeps lightweight identifiers, file paths, queries, links, and loads content on demand. Claude Code takes the second route, using grep and glob over the filesystem rather than a vector index, which sidesteps staleness entirely.

If you are doing classic RAG, Anthropic's contextual retrieval result is the highest-leverage single change available. Before embedding each chunk, prepend fifty to a hundred tokens of model-generated context explaining what that chunk is and where it sits. Do the same before BM25 indexing.

Contextual embeddings−35%

+ contextual BM25−49%

+ reranking−67%

One caveat that saves a lot of engineering. If your knowledge base is under roughly 200,000 tokens, about five hundred pages, skip retrieval and put the whole thing in the prompt behind a cache breakpoint. The retrieval stack is solving a problem you do not have.

Memory means durable state that survives the context window. The foundational work is MemGPT, which borrowed virtual memory from operating systems: a fixed main context plus external recall and archival tiers, with the model paging between them through function calls. That lineage runs through Letta, and alongside it sit A-MEM with its self-linking Zettelkasten notes, MemoryBank with Ebbinghaus-style forgetting curves, Mem0's extract-consolidate-retrieve pipeline, and Zep's bi-temporal knowledge graph, which records both when a fact was true and when the system learned it. That second timestamp is what lets you answer questions about stale information rather than silently serving it.

Sub-agents give each worker its own window and return only a distillation. More on the economics of that below.

Inference layer

Prompt caching is the one you can act on directly, and it is discussed in its own section further down. KV cache eviction methods, StreamingLLM with its attention sinks, H2O's heavy hitters, SnapKV, PyramidKV, matter if you serve your own models and are otherwise inherited invisibly. Prompt compression, principally the LLMLingua family, drops low-information tokens using a small model's perplexity estimates, reaching up to twenty times compression in the original paper.

Model layer

Sparse and linear attention, sliding windows, Infini-attention, recurrent memory transformers, Mamba hybrids, and the position interpolation work that made long context practical at all: linear PI, NTK-aware scaling, YaRN, LongRoPE. Google's Titans line, which learns memory at test time, is the interesting bet here, because if it works some of the scaffolding above becomes unnecessary. Watch it, do not build on it yet.

The application layer is what we spend most of our time on in the AI Engineering Bootcamp, building retrieval, agent loops and evaluation harnesses that hold up outside a demo.

What the vendor features map to

Anthropic's platform features line up against these families cleanly, which is worth spelling out because the marketing names obscure the mechanisms.

• Context editing clears stale tool results server side. This is message-level pruning. Critically it runs after the prompt cache lookup, so it does not invalidate your prefix the way client-side stripping would.
• The memory tool is file-based external storage in a directory you host. This is MemGPT-style paging, productised. When the context nears the clearing threshold the model gets a warning to save anything important first.
• Prompt caching is the economic constraint that shapes everything else.
• Sub-agents are context isolation.
• Skills are dynamic tool loading through progressive disclosure. Only the metadata loads until the skill is relevant, and bundled scripts execute without their source entering the context.

Anthropic's reported numbers for the first two, on internal agentic search evaluations:

Context editing only+29%

Editing + memory+39%

The cache is the budget

Prompt caching gives a ninety percent discount on cached input, at the cost of a 1.25x write. That is the largest single cost lever available, and it imposes one hard architectural constraint: your prefix must be stable.

The cache key is the exact token sequence, in order, up to your breakpoint. Change anything upstream and everything downstream is re-billed at full price. Whitespace counts.

Sub-agents, and what they cost

Anthropic's multi-agent research system uses an orchestrator that plans and spawns three to five parallel workers. Workers never talk to each other and know almost nothing about each other's work. All the coordination logic lives in the orchestrator.

It beat single-agent Claude Opus 4 by 90.2 percent on their internal research evaluation. It also used roughly fifteen times the tokens, and their own analysis found that token usage alone explains eighty percent of the variance in performance.

Tool definitions are context too

An underrated failure mode: the tools themselves. Connect a handful of MCP servers and the definitions alone can consume a large fraction of the window before the conversation starts, while giving the model an ambiguous menu to choose from.

Anthropic's proposed alternative is to present MCP servers as code APIs in a filesystem and let the model write code that calls them, processing intermediate results in the execution environment rather than round-tripping every byte through the context. Their worked example goes from 150,000 tokens to about 2,000, a 98.7 percent reduction. Independent reimplementations report somewhere between 78 and 98 percent, which is a wide band but points the same direction.

A framework

Here is how I would actually decide, keyed to how full the window is rather than to which technique sounds most sophisticated.

And when choosing between the four main moves:

• Compact for linear conversational work that needs continuity.
• Retrieve just in time when the corpus is large or changing and only a slice is relevant per step.
• Offload to files for anything that must survive a clearing pass or a session boundary.
• Spawn sub-agents for parallelisable exploration, accepting the token multiplier.

Measure it, and distrust the leaderboards

None of this is worth doing on faith. Instrument first: track task success rate, tokens per solved task, and cost and latency per run. Context management that does not move those numbers is theatre.

For benchmarks, match the instrument to the question. RULER, HELMET and ∞Bench for raw long-context capability. LoCoMo and LongMemEval for memory across sessions. SWE-bench Verified for agentic long-horizon work.

Then treat published memory-framework comparisons with real suspicion. The frameworks benchmark each other under different judge models and configurations, and the numbers do not survive contact.

Zep, original claim84%

Mem0 recomputation58.44%

Zep rebuttal75.14%

LoCoMo has a further problem worth flagging. Its conversations run to roughly sixteen to twenty-six thousand tokens, which comfortably fits inside a modern window. Scores on it tell you very little about behaviour at the horizons where agents actually operate.

Where this leaves us

The million-token window is real and useful. For single-shot analysis of a whole codebase or a long document it is excellent, you pay the length cost once and rot has limited opportunity to bite. It went generally available on Opus 4.6 and Sonnet 4.6 in March 2026 at flat pricing, with no long-context premium.

If you do intend to use the full window, pick the model deliberately. On the eight-needle MRCR test at one million tokens, Claude Opus 4.6 scores 76 percent mean match ratio against Sonnet 4.5's 18.5 percent on the identical test. Capacity is not the differentiator. Usable capacity is.