← redevops.io  /  Benchmarks  /  Reasoning retrieval · reasoning ↔
Reasoning · measured (in-sample)

Retrieval is necessary, not sufficient

Retrieval and reasoning are orthogonal optimization problems: improving one does not guarantee improving the other. Context Runtime optimizes both — a Retrieval Plane and a Reasoning Plane. This page measures the second: answer quality — how well the model reasons over context it has already been given.

We first ran this deliberately un-optimized and read two datasets as hitting a hard model-capacity wall. Auditing those failures, both were evaluation artifacts — the gold context was truncated below its own length, and one dataset was mislabeled. Corrected, the picture inverts: a single thinking pass over the full context lifts the hardest set from near-zero to 0.83, and the lever is inference-time reasoning — not a bigger model.

The clean measurement is at oracle (gold context only — it isolates reasoning from retrieval). We hold retrieval at gold and vary one knob: direct (a terse extractive pass) vs direct‑think (the same terse prompt with the model's native reasoning on). Qwen3.6-35B, N=24, grok-judged with a liveness-guarded judge:

Qwen3.6-35B · oracle · N=24
Dataset (regime)directdirect-think
PopQAsingle-fact lookup — saturates once any context is present 1.001.00
MuSiQuemulti-hop — chained across passages; thinking hurts here 0.790.71
LongMemEvaltemporal / multi-session — aggregated across sessions 0.170.83
TEMPOtemporal reasoning (rebuilt from TempReason) — point-in-time / duration 0.791.00
NutritionRU domain — rebuilt set ‡ — extractive; thinking is neutral 0.960.92

The ceiling was the eval, not the model LongMemEval oracle read 0.00 in the first run — we called it a capacity wall. It was context truncation: the harness capped oracle at ~12k chars while the gold sessions run 22k–75k, so the model was graded on sessions it was never shown. Fix the cap, turn on thinking: 0.83. The durable lesson is to audit the evaluation before declaring a wall.

Thinking is a per-regime decision, not a global switch Native reasoning lifts the temporal sets (LongMemEval 0.17→0.83, TEMPO 0.79→1.00) but hurts multi-hop (MuSiQue 0.79→0.71) and is neutral on lookup/domain. So the strategy is a learned per-class choice — lookup/multi-hop → direct; temporal/long-memory → direct-think — not "always think."

Size is not the lever — reasoning is A 122B model with no thinking scores LongMemEval oracle 0.25 — below the 35B with thinking (0.83), and no better than the 35B without it. Parameter count does not buy cross-session aggregation; inference-time reasoning over the full context does. Model selection stays competence-driven per class, not size-driven.

No single retriever wins every regime — so route On retrieved context the best method changes by regime — DIVER on multi-hop, BM25 on long-memory, hybrid on temporal — so a policy that routes the (retriever, strategy) per regime edges the best single method: routed 0.86 vs DIVER 0.85, hybrid 0.83, ReasonIR 0.81, BM25 0.81 (N=24). The margin is modest and in-sample (the route table is built from this same cube) and its absolute value leans on the rebuilt nutrition regime ‡; the durable point is that no fixed retriever is best everywhere. True generalization is the online bandit on a held-out stream.

Measurement discipline These numbers are only trustworthy because the judge is verified live before every run — an earlier investigation was corrupted when the judge silently went down and floored every score to a substring-match rate. The retrieval page describes the liveness guard that now fails loud instead.

Honest scope Oracle, Qwen3.6-35B, N=24, grok-judged. In-sample and modest by design — the claim is "no fixed retriever wins every regime, and thinking beats size," not a large end-to-end number. Public datasets carry a contamination caveat (PopQA closed-book ≈ 0.5). ‡ Re-run 2026-07: the datasets were re-measured at N=24 after a storage cleanup wiped the working copies — PopQA / MuSiQue / LongMemEval / TEMPO reproduced the earlier oracle numbers within judge noise, but the RU nutrition set could not be recovered and was rebuilt from source, so its numbers are a fresh baseline, not comparable to earlier nutrition figures.

// PopQA · MuSiQue · LongMemEval · TEMPO · Nutrition (RU) · oracle · direct vs direct-think · Qwen3.6-35B N=24 · liveness-guarded grok judge

Retrieval choice reaches the answer

Oracle isolates reasoning; the real system must retrieve first. Here each method's retrieved context (its recall is on the retrieval page) is passed to the same Qwen3.6-35B answerer — best strategy per regime — and we score the answer, end to end (N=24):

Answer accuracy · retrieved context · Qwen3.6-35B · N=24mean
RetrieverPopQAMuSiQueLongMemTEMPONutrition ‡—
BM25lexical baseline 1.000.460.830.751.000.81
hybriddense (bge) + BM25 → RRF 1.000.540.710.881.000.83
ReasonIRreasoning-tuned embedder 1.000.420.750.881.000.81
DIVERLLM expand → rerank 1.000.710.750.880.920.85
CR-autorouted retriever + strategy per regime 1.000.620.830.831.000.86

Retrieval quality survives to the answer The multi-hop spread from the recall matrix reappears in answer accuracy — on MuSiQue, DIVER's stronger retrieval reads 0.71 vs ReasonIR's 0.42. A better passage in the window is a better answer out: retrieval is necessary.

The retrieval → answer gap Against oracle (gold context), imperfect retrieval costs most where the passage is hardest to find and reason over — TEMPO −0.12, MuSiQue −0.08, LongMemEval −0.04, PopQA none. Closing that gap is exactly the Retrieval Plane's job; here it is measured, not assumed.

...and no fixed retriever is best everywhere DIVER tops multi-hop, BM25/CR-auto top long-memory, hybrid/ReasonIR top temporal — so CR-auto routes per regime and posts the best mean (0.86). Modest and in-sample; the absolute means lean on the rebuilt nutrition regime (‡, see below).

// answer accuracy · retrieved context · best strategy per regime · Qwen3.6-35B · N=24 · liveness-guarded grok judge · reproduce: redevops_rag/benchmarks/eval_cube2.py

A stronger answerer is competence-routed too

"Size is not the lever" above uses a no-think 122B. Here we swap in a genuine frontier reasoner — Kimi-k2.6, native chain-of-thought — against the 35B, best method+strategy per regime. Capacity is not a global win: it helps where the task is aggregation and hurts where it is composition.

oracle (gold ctx)retrieved
Dataset (regime)Qwen-35BKimi-k2.6Qwen-35BKimi-k2.6
MuSiQuemulti-hop composition — the 35B wins 0.790.670.710.67
LongMemEvalcross-session aggregation — the frontier helps 0.880.880.830.88
TEMPOtemporal — frontier helps at retrieved 1.001.000.880.92
Nutrition ‡RU domain — rebuilt set 0.960.961.000.96
meanPopQA 1.00 both — omitted 0.930.900.880.88

Capacity helps aggregation, not composition The frontier reasoner lifts the temporal / long-memory sets at retrieved (LongMemEval 0.83→0.88, TEMPO 0.88→0.92) but is worse on multi-hop (MuSiQue 0.71→0.67), netting even overall (0.88 vs 0.88). Cross-session aggregation rewards a deeper reasoner; composition over clean passages does not.

So the answerer is routed, not maximized A stronger model is one more per-class competence, not a global upgrade — the runtime selects it where the regime rewards it (temporal), the same routing logic as retriever and strategy. Buying the frontier model everywhere would lose multi-hop accuracy and pay the API cost for it. Competence-driven, not size-driven.

// answer accuracy · Qwen3.6-35B vs Kimi-k2.6 · best method+strategy per regime · N=24 · liveness-guarded grok judge

Context size is its own knob

How much retrieved context to pass the model is a reasoning-side decision, separate from retrieval quality. Passing more is not monotonically better — the classic lost-in-the-middle tail sets in (LongMemEval-S, temporal / multi-session, grok-judged accuracy):

Context passedQwen3.6-35BDeepSeek-Flash · Q4 CPU
~3.5kone full session — the floor 0.1670.083
~4.5k~1.5 sessions 0.2220.250
~10.5k~3 sessions 0.3330.167

There is a sweet spot More context lifts Qwen off the floor (0.167 → 0.333 as coverage grows ~3.5k → ~10.5k), but DeepSeek peaks near ~4.5k (0.250) then dips at ~10.5k (0.167) — the lost-in-the-middle tail. So the runtime should target a band per model/class, not dump everything, which is what a reward that nets accuracy against token cost learns.

Model-dependent The optimal band differs by model, so context sizing is a per-mission-class reasoning decision — not a global constant.

// LongMemEval-S · temporal / multi-session · grok-judged · reproduce: redevops_rag/benchmarks/eval_v5_context_size.py

Data sources & licensing
These datasets are used only to develop and benchmark the Context Runtime software itself - they are not redistributed, not embedded in the product, and not part of any customer deployment. Context Runtime customers apply it to their own data under their own licenses. Each dataset is used under its own license; loaders download it on demand.
  • PopQA - MIT.
  • MuSiQue - CC-BY-4.0.
  • LongMemEval - MIT.
  • TempReason (shown as TEMPO) - CC-BY-SA-3.0.
  • Nutrition (RU) - a proprietary internal corpus (third-party lecture content) with no open license; reported as aggregate recall only and not redistributed, used as an anti-contamination control.
redevops.io is a member of the NVIDIA Inception program.
© 2026 NVIDIA, the NVIDIA logo, and NVIDIA Inception are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries.