← redevops.io  /  Benchmarks  /  Retrieval ↔ Reasoning
Retrieval · reproducible

Retrieval benchmarks

Retrieval and reasoning are orthogonal optimization problems; this is the retrieval half - can the runtime find the right evidence? One seeded, ground-truth simulation run in the Python source-of-truth and the Go port (v1→v2 calibration), then the forward axes: drift adaptation, knowledge routing with the per-regime retrieval-quality matrix, and the encoder A/B. The reasoning half - can the model use it? - is measured separately.

Sections: drift · knowledge routing & retrieval matrix · property-graph routing (real KYC data) · encoders ↓

v1→v2 · seeded simulation

Calibration, abstention & the sizer

A seeded, synthetic environment - not a live dataset - that isolates the v1→v2 mechanism: the reward finally sees calibrated relevance, abstention turns on, and the survival-sizer prunes the low-value tail. The real-dataset results are the retrieval matrix and encoder A/B further down; this section is a controlled simulation of the mechanism, not a corpus benchmark.

Python - source of truthGo - port
Metricv1v2Δv1v2Δ
Learned-policy precision Share of served passages that are truly relevant, once the policy converges. v2's reward finally sees calibrated relevance instead of a coverage-biased judge.
py v1 67.6v2 82.2
67.6%82.2%▲ +14.6 84.6%95.9%▲ +11.3
Abstention recall Truly-unanswerable queries that v2 declines to answer. v1 has no abstention at all - it always answers.
py v1 0v2 100
0.0%100%▲ +100 0.0%100%▲ +100
False-abstain rate Answerable queries v2 wrongly declined - the cost of abstention. It stays at zero: the floor never fires on a good query. 0.0%0.0%- none 0.0%0.0%- none
Expensive-stage depth Passages sent to the costly rerank / synthesis stage from a deep k=8 arm. The survival-product sizer prunes the low-relevance tail before it is paid for.
py v1 8.0v2 3.0
8.003.00▼ −62% 8.003.00▼ −63%
Precision after the sizer Precision of what survives the gate. The passages the sizer removed were the wrong ones, so precision rises to 100%.
py v1 37.5v2 100
37.5%100%▲ +62.5 37.5%100%▲ +62.5
v1 baseline v2 · calibration · abstention · sizer ▲ improvement

Honest by construction The v2 additions are opt-in, so v2-with-flags-off is byte-for-byte v1 - the A/B is a flag toggle in one process, not two codebases.

40-seed average A single run is dominated by bandit exploration luck, so precision is averaged over 40 seeds; every number here is deterministic and reproducible.

β = 0.9 headline Precision is shown at the calibration-trust ceiling (shipped default 0.5); the lift grows monotonically with how much the reward trusts calibrated relevance.

Two runtimes, one methodology Absolute precision differs because each runtime uses its own strategy set - but both show a double-digit lift, full abstention, and a ~62% depth cut.

// ground-truth simulation · 600-query stream · precision measured on answerable, answered queries only · reproduce: github.com/redevops-io/context-runtime

v3 · seeded simulation

Online optimization under drift

A forward-looking axis, distinct from the shipped v1→v2 calibration above - and, like it, a seeded synthetic simulation (a two-arm environment with hand-set rewards), not a corpus benchmark. The best plan drifts mid-run - a model upgrade, a corpus shift. A static v1/v2 planner stays pinned to the now-stale plan; the v3 online planner re-explores, and recency-weighted (discounted) learning lets it track the shift. We report the post-drift average served-plan reward, seed-averaged. It isolates the adaptation mechanism; it is not measured on the datasets below.

Metricv2 (static)v3 (online)Δ
Post-drift served-plan reward Average reward on the served plan after the environment drifts. Discounting lets stale evidence fade, so the planner tracks the shift and recovers most of the oracle. 0.200.67▲ +0.47
Post-drift reward, online without discounting Plain sample-average online learning adapts poorly - 200 pre-drift observations drown the new signal. 0.200.34+0.14

Discounting is the mechanism Recency-weighted learning is what converts online adaptation from a marginal gain (+0.14) into recovery of most of the oracle reward (0.67 of a 0.80 post-drift ceiling).

Orthogonal to v2 This axis is distinct from - and preserves - the v1→v2 calibration gains above; it measures adaptation in a non-stationary world, not steady-state precision.

Preliminary A seeded drift simulation, kept separate from the shipped results until promoted. The numbers are deterministic and regenerable.

// seeded drift simulation · 24-seed average · post-drift oracle = 0.80 · reproduce: python examples/online_vs_static_bench.py

v4 · knowledge routing · measured on real datasets

From retrieval tuning to knowledge planning

v1-v3 tune how to retrieve; v4 first decides what representation the question needs - a flat document index, a multi-hop entity graph, or temporal memory - then routes there. Three public datasets, each built to stress one regime: PopQA (single-fact lookup), MuSiQue (multi-hop), LongMemEval (temporal). Routing accuracy grades the label against the regime each was built for.

A learned router reads implicit intent; a keyword router is blind to it (routing accuracy per regime):

Query regimeKeyword routerLLM router (Qwen3.6-27B)Δ
PopQA → documentsingle-fact lookup - lexical cues are enough, so both routers land it 0.951.00+0.05
MuSiQue → graphmulti-hop - no keyword says "this needs a graph"; the rule router never routes it 0.000.90▲ +0.90
LongMemEval → temporalconversational memory - implicit recency intent, invisible to keywords 0.001.00▲ +1.00

…and once routed, the retrieval-quality matrix shows why - every method across every regime (recall@4 vs gold, 15 distractors/question). Where a regime separates retrievers, each wins a different one and the routed policy (CR-auto) tracks the column winner; where it saturates, the differentiation moves to the answer side:

Methodpopqa · lookupmusique · multi-hoplongmemeval · temporaltempo · temp-reasonnutrition · RU domain
BM25lexical baseline 1.000.600.840.590.87
hybriddense (bge) + BM25 → RRF 1.000.600.770.590.80
ReasonIRreasoning-tuned 8B embedder 1.000.670.810.590.87
DIVERLLM expand → hybrid → listwise rerank 1.000.930.890.590.67
SimGraphdependency-free 2-hop graph 1.000.200.340.590.07
HippoRAGLLM-OpenIE entity graph + PPR 1.000.73n/a0.59n/a
Graphitibi-temporal Neo4j KG n/an/a0.23‡0.46n/a
CR-autothe learned routed policy 1.000.970.880.590.87

Multi-hop is where retrieval separates - and routing wins it On MuSiQue the reasoning-retrieval and graph arms pull clear of lexical (DIVER 0.93, HippoRAG 0.73 vs BM25/hybrid 0.60, SimGraph 0.20); CR-auto routes to the winner and lands 0.97. LongMemEval separates the same way (DIVER 0.89 vs SimGraph 0.34). This is the regime where "the right retriever per query" is worth the most.

Temporal-reasoning does not separate at retrieval - it's an answer-side problem On TEMPO every method scores the same 0.59 (recall@10: 0.885): the gold passage is easy to find, so retrieval choice is moot. What separates models here is reasoning over the passage - where a thinking pass lifts TEMPO from 0.79 to 1.00 (see reasoning). We retired an earlier "graph engines collapse on TEMPO" claim: it was measured on a mislabeled dataset, and on the corrected one no method collapses.

Graph engines: competitive, not dominant - and heavy ones have an ingestion ceiling HippoRAG's entity graph is a real multi-hop arm (0.73) but reasoning-retrieval beats it here; SimGraph, the dependency-free approximation, trails badly off lookup. Graphiti (bi-temporal KG) - once a bounded-context fork let it ingest without overflowing the 32k window - lands worst on both temporal regimes it targets: LongMemEval 0.23 (‡ on a reduced n=8 - 7 of 15 items failed the LLM edge-extraction) and TEMPO 0.46 (below the 0.59 every other method ties at), at ~10 hours per run. The heaviest infrastructure here is the slowest, the least reliable, and the weakest - a stark case that heavy graph machinery must earn its cost per regime, and here it doesn't. Exactly the call the planner makes. We also ran LightRAG - the most popular off-the-shelf graph-RAG - through the same corpora: it placed well behind HippoRAG in every regime we tested, so among graph retrievers HippoRAG is the one the runtime defaults to.

The regime flips on Russian DIVER drops to 0.67 on the Russian domain corpus - its English LLM query-expansion misfires - while the multilingual encoder leads (ReasonIR 0.87 at recall@4, 0.93 at recall@10, and it wins ranking outright - see encoders). SimGraph's English entity graph collapses on Russian (0.07). This is why the runtime selects the encoder per corpus.

recall@4 headline, recall@10 in view We report recall@4 because it is discriminative; at recall@10 the easier columns saturate (MuSiQue DIVER 1.00, tempo 0.885 across the board, nutrition ReasonIR 0.93 pulling clear of BM25) - the ranking edge, not just presence, is what reaches the model. Deterministic, reproducible; nutrition is proprietary (aggregate recall only).

// PopQA · MuSiQue · LongMemEval · TEMPO · Nutrition (RU) · methods: BM25 / hybrid / ReasonIR / DIVER / SimGraph / HippoRAG / Graphiti / CR-auto · recall@4 vs gold, N=15, 15 distractors/q · ‡ Graphiti ran via a bounded-context fork of the 32k-window extractor; LongMemEval is n=8 (7/15 items failed LLM edge-extraction) and ~10h/run · reproduce: redevops_rag/benchmarks/eval_matrix.py

v6 · property-graph routing · measured on real KYC data

A fifth engine: deterministic ownership traversal

v4 routes between document, a reasoning graph and temporal memory. Ownership questions - "who ultimately owns X" - need none of those: they need an exact, auditable traversal over a curated property graph, the system-of-record complement to HippoRAG's reasoning graph. We added cypher / gremlin methods to the graph representation (they coexist with HippoRAG) and measured the whole thing on real public data: 182,445 GLEIF entities · 168,818 ownership edges · 99.99% named · 3,765 sanctioned (OpenSanctions), loaded into Neo4j.

Given a plain-English request, the planner now selects across five engines - two of them (HippoRAG and the property graph) sharing the graph representation but answering different questions (selected-engine accuracy per query shape, 59 labelled KYC/KYB queries):

Query shape → engine the planner picksaccuracy
"how is X connected to the sanctioned network" → HippoRAGmulti-hop reasoning - the connections are the answer100%
"sanctions case reference OFAC-56789" → BM25an exact identifier - lexical match, not dense paraphrase100%
"ownership structure as of 2021" → temporal (Graphiti)point-in-time state100%
"who ultimately owns Acme Corp" → property graph (Cypher)deterministic ownership traversal - the new engine100%
"how many sanctioned entities per country" → SQLan aggregate - only the DB retriever can compute it80%
"what kind of company is X" → dense vectorssemantic / conceptual - embeddings, not keywords75%
representation routing 94.9% selected-engine 91.5% overall, across all 59 queries

…and once routed, the traversal is what property graphs are for: the variable-depth ownership walk is sub-millisecond on 182k entities. The only real cost is finding the seed by name - which an index removes:

Ownership query on the 182,445-entity graph (Neo4j)p50 latency
the N-hop traversal itselffrom an indexed seed - index-free adjacency, every hop a pointer walk0.53 ms
name → ultimate owner · CONTAINS scanthe portable default - scans all 182k entities to find the seed67 ms
name → ultimate owner · full-text indexindexed seed lookup + the same traversal7 ms

Two graphs, chosen apart HippoRAG (reasoning) and the property graph (system of record) are backends of the same representation; the planner sends "how is X connected" to the first and "who owns X" to the second - approximate reasoning where it helps, exact ownership where it must be exact.

The traversal was never the cost 0.53 ms for the full variable-depth walk on a six-figure graph. The 67 ms was the seed scan; a full-text index makes name → ultimate-owner 7 ms - comfortably inside a millisecond-class Go/No-Go budget.

Real chains, real hits AGC Automotive Poland → … → AGC Inc (Japan); a sanctioned entity consolidated by UniCredit; DB USA under Deutsche Bank. The retriever returns the identical owner set as a ground-truth Cypher walk, and is injection-safe against the live DB.

Honest misroutes 91.5%, not a fitted 100%: "across" is a multi-hop trigger that pulls an aggregate onto the graph path; "ownership" as an abstract noun over-fires the graph hint on definitional questions. Legible failures, each naming its fix.

// GLEIF golden-copy (relationship records + LEI-CDF names) + OpenSanctions default targets · 182,445 entities / 168,818 ownership edges in Neo4j · routing over 59 labelled KYC/KYB queries · latency p50 over 300 real ownership traversals · reproduce: CR-enterprise/benchmarks/graph-routing/

Encoders · validated

Better embeddings lift retrieval further

Everything above shows retrieval was already strong before any reasoning-side optimization. One input from that work - embedding quality - feeds straight back into retrieval. We A/B'd three encoders on a proprietary Russian-language, domain corpus (nutrition), scored against gold chunks. Nemotron-3-Embed-8B (RTEB #1) wins both recall and ranking:

Encoderrecall@10NDCG@10
bge-small-en · 384-dthe cheap CPU default 0.6500.441
ReasonIR-8Breasoning-tuned 8B embedder 0.6000.505
Nemotron-3-Embed-8BRTEB #1 · multilingual · 4096-d 0.6630.549

Ranking is what matters Only the top chunks reach the model, so NDCG@10 is the number that counts - Nemotron is +0.108 over bge (0.441 → 0.549), and also edges recall (0.663 vs 0.650). RTEB #1 validated on our own data, not a public leaderboard.

Regime-dependent, so the runtime routes it On English reasoning-intensive retrieval the cheap encoder wins (DIVER dominates ranking); on domain / multilingual corpora a stronger encoder wins. Context Runtime selects the encoder per corpus - Nemotron-Embed is the default for domain / multilingual retrieval, bge stays the cheap default elsewhere.

Chunk-level, and now answer-valid Scored at chunk level - 8,775 × 1200-char chunks, matched by gold-passage text-overlap (chunk-scheme-agnostic). The gold passage now surfaces ~0.65 in the top 10, versus a truncated whole PDF before - so this domain is retrieval-valid for the reasoning benchmark too.

Multimodal path (prototype) Rendering the source PDFs to page images and retrieving with CLIP scores 0.525 (doc-level) - below text retrieval on this text-dense corpus, since generic CLIP is trained on natural images, not lecture pages. A document-visual model (ColPali / ColQwen) is the next step for the diagram/scheme questions where the answer is visual; the render → embed → retrieve pipeline is already in place.

The point Retrieval performed very well before reasoning optimization; optimizing embedding quality only lifted it further. The two planes compound - they do not trade off.

// proprietary nutrition corpus (RU) · chunk-level (8,775 × 1200-char) · recall@10 / NDCG@10 vs gold passages · + a CLIP page-image path (0.525 doc-level) · reproduce: redevops_rag/benchmarks/eval_embedders.py

Retrieval is half the story. See Reasoning benchmarks → for whether the model can use what retrieval finds.
Data sources & licensing
These datasets are used only to develop and benchmark the Context Runtime software itself - they are not redistributed, not embedded in the product, and not part of any customer deployment. Context Runtime customers apply it to their own data under their own licenses. Each dataset is used under its own license; loaders download it on demand.
  • PopQA - MIT.
  • MuSiQue - CC-BY-4.0.
  • LongMemEval - MIT.
  • TempReason (shown as TEMPO) - CC-BY-SA-3.0.
  • GLEIF golden-copy - CC0 1.0; OpenSanctions - CC-BY-NC 4.0, non-commercial research use only (commercial/compliance deployment needs an OpenSanctions license).
  • Nutrition (RU) - a proprietary internal corpus (third-party lecture content) with no open license; reported as aggregate recall only and not redistributed, used as an anti-contamination control.
redevops.io is a member of the NVIDIA Inception program.
© 2026 NVIDIA, the NVIDIA logo, Nemotron, and NVIDIA Inception are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries.