Answers From Less Context

A context window is a cliff. The obvious fix for long-context work is a bigger-context model: pour the whole history in and let attention sort it out. But every model has a window, and past it the answer-bearing evidence simply isn't in the prompt anymore. Accuracy falls off. The runtime's job is to decide what enters the model, so a small, cheap, self-hostable window can answer over an arbitrarily large corpus.
This is the follow-up to Better Context Beats More Context. There we showed quality beats quantity at a fixed scale. Here we push the scale, from ten thousand to a million tokens, and ask a sharper question: when the corpus is far larger than the window, does managing context still win, and by how much?
The design: decide, then materialize
Retrieval finds candidates; the runtime decides what becomes context. We added a cheap decision stage in front of the answerer: narrow to the relevant regions, then materialize only as deeply as the question requires:
corpus (up to 1M tokens)
→ region routing cheap deterministic scorer → top-K regions
→ materialize depth state → sparse → deep → full, escalate only if needed
→ fixed-window answerer ~4k tokens in
Underneath sit a reproducible context state with hash-pinned evidence identity, an additive prior that warm-starts the existing bandit for familiar contexts, and per-tenant cache isolation. Every layer respects the tenant boundary. None of it is exotic. It is the difference between giving the model everything and deciding what the model should see.
The test: real model, real haystacks, deterministic scoring
- Answerer:
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4: a hybrid Mamba-2 + MoE model, 3B active, 128k context window, served on a single 32GB RTX PRO 4500 Blackwell. It is a plain fixed-window consumer; it does not manage its own context, so the result is attributable to the runtime. - Corpora: two independent Wikimedia sets: StrategyWiki (development) and Simple English Wikipedia (held-out validation). Using a second corpus guards against tuning the harness to its own test set.
- Task: a needle in a real haystack. The haystack is real Wikipedia; the needle is a synthetic sentence carrying a random facility id and a random access code, dropped in at a controlled depth. The question asks for that code by its id. Because the code is randomly minted at runtime, it appears in no training set: the model can only read it. Turning the corpus up to a million tokens buries the same needle in ever more hay. Scoring is a deterministic substring match: no LLM judge, cheap and reproducible at a million tokens.
Two arms carry the comparison: full feeds the whole haystack truncated to the window; managed feeds only what context management retrieves (~4k tokens).
The result: full context collapses, managed context holds
Accuracy and the tokens fed to the answerer, per corpus:
StrategyWiki (development)
| corpus size | full context | managed context | tokens: full → managed |
|---|---|---|---|
| 120k | 1.0 | 1.0 | 111k → 4.2k |
| 500k | 0.25 | 1.0 | 111k → 4.2k |
| 1M | 0.0 | 1.0 | 111k → 4.2k |
Simple English Wikipedia (held-out)
| corpus size | full context | managed context | tokens: full → managed |
|---|---|---|---|
| 120k | 0.5 | 0.5 | 111k → 4.1k |
| 500k | 0.0 | 0.75 | 111k → 4.1k |
| 1M | 0.0 | 1.0 | 111k → 4.1k |
Past the window, dumping the whole corpus in falls to zero on both corpora. The needles are outside the prompt. Managed context answers from about 4,100 tokens, roughly 27× fewer than full's 111k, and stays high. Latency tracks tokens: full runs ~5–9 s at long horizons, managed ~2.5–3 s. This is the "better answers from less context, and smaller self-hosted models made viable" result, now with a real 128k model instead of a simulated one, and it reproduces on the independent corpus.
Is the model just remembering Wikipedia?
No, by construction, and the data proves it. The answer is a random code that exists in no training set. And the collapse itself is the proof: if the model were answering from memory, full context would not fall to zero as the corpus grows. The answer would be equally (un)knowable either way. Accuracy tracking whether the needle is inside the window is exactly what reading-from-context looks like.
Would a bigger model change this?
Split the chart at the window. Within the window, a stronger model would likely clean up the pre-collapse dips: finding a needle in 100k tokens is genuinely capacity-sensitive, and this is a small 3B-active model. Past the window, no. The needle has been truncated out of the prompt, so any model with the same window still scores zero: the wall is the window at any parameter count. And the managed arms are retrieval-bound: once the code is in context, extracting it is trivial, so extra capacity barely helps. A bigger model tidies the left of the chart; it does not move the frontier. The small model already saturates the managed arms, which is the point.
What we are not claiming
- The trend is robust; individual cells are not. With four needles per point, accuracy is quantized to 0.25 and carries real-model nondeterminism. More needles would sharpen the estimates. They would not change the direction.
- The win is the layer itself. The decisive result is that managed context beats full context past the window. The finer optimizations (sparse region routing, adaptive depth, learned priors) are additive refinements measured against that baseline, and at this scale sparse routing mostly ties the plain-retrieval baseline. Each must keep earning its place.
- The held-out corpus is genuinely harder. Simple English Wikipedia's list-heavy articles give a noisier retrieval signal, so managed accuracy there runs 0.5–1.0, missing a clean 1.0. Full still collapses to zero while managed holds well above it.
The verdict
Adopt managed context as the default for long-horizon work: past the window it is the difference between an answer and nothing, at a fraction of the cost. But keep the discipline that made the result trustworthy: the optimizer must remain free to pour everything in when the corpus fits the window and that wins; the finer optimizations ship additive and gated, each earning its place against a frozen baseline, free of fiat.
The optimizer is successful only when it chose better than the alternatives, including doing nothing and giving the model full context. That is the whole design, and it is now source-available: the context-management layer lives in the Context Runtime core (AGPL-3.0 with the Commons Clause), with every result stamped by the exact commit and source hash of the code that produced it.