Retrieval and reasoning are orthogonal optimization problems: improving one does not guarantee improving the other. Context Runtime optimizes both — a Retrieval Plane and a Reasoning Plane. This page measures the second: answer quality — how well the model reasons over context it has already been given.
We first ran this deliberately un-optimized and read two datasets as hitting a hard model-capacity wall. Auditing those failures, both were evaluation artifacts — the gold context was truncated below its own length, and one dataset was mislabeled. Corrected, the picture inverts: a single thinking pass over the full context lifts the hardest set from near-zero to 0.83, and the lever is inference-time reasoning — not a bigger model.
The clean measurement is at oracle (gold context only — it isolates reasoning from retrieval). We hold retrieval at gold and vary one knob: direct (a terse extractive pass) vs direct‑think (the same terse prompt with the model's native reasoning on). Qwen3.6-35B, N=24, grok-judged with a liveness-guarded judge:
| Qwen3.6-35B · oracle · N=24 | ||
|---|---|---|
| Dataset (regime) | direct | direct-think |
| PopQAsingle-fact lookup — saturates once any context is present | 1.00 | 1.00 |
| MuSiQuemulti-hop — chained across passages; thinking hurts here | 0.79 | 0.71 |
| LongMemEvaltemporal / multi-session — aggregated across sessions | 0.17 | 0.83 |
| TEMPOtemporal reasoning (rebuilt from TempReason) — point-in-time / duration | 0.79 | 1.00 |
| NutritionRU domain — rebuilt set ‡ — extractive; thinking is neutral | 0.96 | 0.92 |
The ceiling was the eval, not the model LongMemEval oracle read 0.00 in the first run — we called it a capacity wall. It was context truncation: the harness capped oracle at ~12k chars while the gold sessions run 22k–75k, so the model was graded on sessions it was never shown. Fix the cap, turn on thinking: 0.83. The durable lesson is to audit the evaluation before declaring a wall.
Thinking is a per-regime decision, not a global switch Native reasoning lifts the temporal sets (LongMemEval 0.17→0.83, TEMPO 0.79→1.00) but hurts multi-hop (MuSiQue 0.79→0.71) and is neutral on lookup/domain. So the strategy is a learned per-class choice — lookup/multi-hop → direct; temporal/long-memory → direct-think — not "always think."
Size is not the lever — reasoning is A 122B model with no thinking scores LongMemEval oracle 0.25 — below the 35B with thinking (0.83), and no better than the 35B without it. Parameter count does not buy cross-session aggregation; inference-time reasoning over the full context does. Model selection stays competence-driven per class, not size-driven.
No single retriever wins every regime — so route On retrieved context the best method changes by regime — DIVER on multi-hop, BM25 on long-memory, hybrid on temporal — so a policy that routes the (retriever, strategy) per regime edges the best single method: routed 0.86 vs DIVER 0.85, hybrid 0.83, ReasonIR 0.81, BM25 0.81 (N=24). The margin is modest and in-sample (the route table is built from this same cube) and its absolute value leans on the rebuilt nutrition regime ‡; the durable point is that no fixed retriever is best everywhere. True generalization is the online bandit on a held-out stream.
Measurement discipline These numbers are only trustworthy because the judge is verified live before every run — an earlier investigation was corrupted when the judge silently went down and floored every score to a substring-match rate. The retrieval page describes the liveness guard that now fails loud instead.
Honest scope Oracle, Qwen3.6-35B, N=24, grok-judged. In-sample and modest by design — the claim is "no fixed retriever wins every regime, and thinking beats size," not a large end-to-end number. Public datasets carry a contamination caveat (PopQA closed-book ≈ 0.5). ‡ Re-run 2026-07: the datasets were re-measured at N=24 after a storage cleanup wiped the working copies — PopQA / MuSiQue / LongMemEval / TEMPO reproduced the earlier oracle numbers within judge noise, but the RU nutrition set could not be recovered and was rebuilt from source, so its numbers are a fresh baseline, not comparable to earlier nutrition figures.
// PopQA · MuSiQue · LongMemEval · TEMPO · Nutrition (RU) · oracle · direct vs direct-think · Qwen3.6-35B N=24 · liveness-guarded grok judge
Oracle isolates reasoning; the real system must retrieve first. Here each method's retrieved context (its recall is on the retrieval page) is passed to the same Qwen3.6-35B answerer — best strategy per regime — and we score the answer, end to end (N=24):
| Answer accuracy · retrieved context · Qwen3.6-35B · N=24 | mean | |||||
|---|---|---|---|---|---|---|
| Retriever | PopQA | MuSiQue | LongMem | TEMPO | Nutrition ‡ | — |
| BM25lexical baseline | 1.00 | 0.46 | 0.83 | 0.75 | 1.00 | 0.81 |
| hybriddense (bge) + BM25 → RRF | 1.00 | 0.54 | 0.71 | 0.88 | 1.00 | 0.83 |
| ReasonIRreasoning-tuned embedder | 1.00 | 0.42 | 0.75 | 0.88 | 1.00 | 0.81 |
| DIVERLLM expand → rerank | 1.00 | 0.71 | 0.75 | 0.88 | 0.92 | 0.85 |
| CR-autorouted retriever + strategy per regime | 1.00 | 0.62 | 0.83 | 0.83 | 1.00 | 0.86 |
Retrieval quality survives to the answer The multi-hop spread from the recall matrix reappears in answer accuracy — on MuSiQue, DIVER's stronger retrieval reads 0.71 vs ReasonIR's 0.42. A better passage in the window is a better answer out: retrieval is necessary.
The retrieval → answer gap Against oracle (gold context), imperfect retrieval costs most where the passage is hardest to find and reason over — TEMPO −0.12, MuSiQue −0.08, LongMemEval −0.04, PopQA none. Closing that gap is exactly the Retrieval Plane's job; here it is measured, not assumed.
...and no fixed retriever is best everywhere DIVER tops multi-hop, BM25/CR-auto top long-memory, hybrid/ReasonIR top temporal — so CR-auto routes per regime and posts the best mean (0.86). Modest and in-sample; the absolute means lean on the rebuilt nutrition regime (‡, see below).
// answer accuracy · retrieved context · best strategy per regime · Qwen3.6-35B · N=24 · liveness-guarded grok judge · reproduce: redevops_rag/benchmarks/eval_cube2.py
"Size is not the lever" above uses a no-think 122B. Here we swap in a genuine frontier reasoner — Kimi-k2.6, native chain-of-thought — against the 35B, best method+strategy per regime. Capacity is not a global win: it helps where the task is aggregation and hurts where it is composition.
| oracle (gold ctx) | retrieved | |||
|---|---|---|---|---|
| Dataset (regime) | Qwen-35B | Kimi-k2.6 | Qwen-35B | Kimi-k2.6 |
| MuSiQuemulti-hop composition — the 35B wins | 0.79 | 0.67 | 0.71 | 0.67 |
| LongMemEvalcross-session aggregation — the frontier helps | 0.88 | 0.88 | 0.83 | 0.88 |
| TEMPOtemporal — frontier helps at retrieved | 1.00 | 1.00 | 0.88 | 0.92 |
| Nutrition ‡RU domain — rebuilt set | 0.96 | 0.96 | 1.00 | 0.96 |
| meanPopQA 1.00 both — omitted | 0.93 | 0.90 | 0.88 | 0.88 |
Capacity helps aggregation, not composition The frontier reasoner lifts the temporal / long-memory sets at retrieved (LongMemEval 0.83→0.88, TEMPO 0.88→0.92) but is worse on multi-hop (MuSiQue 0.71→0.67), netting even overall (0.88 vs 0.88). Cross-session aggregation rewards a deeper reasoner; composition over clean passages does not.
So the answerer is routed, not maximized A stronger model is one more per-class competence, not a global upgrade — the runtime selects it where the regime rewards it (temporal), the same routing logic as retriever and strategy. Buying the frontier model everywhere would lose multi-hop accuracy and pay the API cost for it. Competence-driven, not size-driven.
// answer accuracy · Qwen3.6-35B vs Kimi-k2.6 · best method+strategy per regime · N=24 · liveness-guarded grok judge
How much retrieved context to pass the model is a reasoning-side decision, separate from retrieval quality. Passing more is not monotonically better — the classic lost-in-the-middle tail sets in (LongMemEval-S, temporal / multi-session, grok-judged accuracy):
| Context passed | Qwen3.6-35B | DeepSeek-Flash · Q4 CPU |
|---|---|---|
| ~3.5kone full session — the floor | 0.167 | 0.083 |
| ~4.5k~1.5 sessions | 0.222 | 0.250 |
| ~10.5k~3 sessions | 0.333 | 0.167 |
There is a sweet spot More context lifts Qwen off the floor (0.167 → 0.333 as coverage grows ~3.5k → ~10.5k), but DeepSeek peaks near ~4.5k (0.250) then dips at ~10.5k (0.167) — the lost-in-the-middle tail. So the runtime should target a band per model/class, not dump everything, which is what a reward that nets accuracy against token cost learns.
Model-dependent The optimal band differs by model, so context sizing is a per-mission-class reasoning decision — not a global constant.
// LongMemEval-S · temporal / multi-session · grok-judged · reproduce: redevops_rag/benchmarks/eval_v5_context_size.py
The Reasoning Plane makes each of these a learned decision, keyed by mission class: the reasoning strategy (direct for lookup/multi-hop; direct-think for temporal/long-memory — measured above), the context size (target a band, not a dump — below), self-verification and retry (where it pays off), and the encoder (a stronger embedder for domain/multilingual — see Retrieval → encoders). Only deterministic strategies ship: an ablation of prompt-scaffolded reason/decompose/map-reduce arms did not reproduce and was dropped.
The routing win above is an in-sample ceiling — the route table was built from the same cube. The next step is the online bandit: learn the policy on a live query stream and evaluate it on a disjoint held-out stream — the honest generalization test. Early held-out runs show the online policy out-generalizing a static table while remaining sample-hungry; a robust-default, confidence-gated shape is what makes it safe to ship.