Why Context Runtime Started Routing Knowledge Instead of Retrieving It

Most RAG systems assume there is a single best knowledge representation. Our benchmarks challenged that assumption.
We tested three real retrieval engines, document (BM25/hybrid), graph (HippoRAG), and temporal (Graphiti), across three reasoning regimes. The runtime's first task was deciding which knowledge representation should answer it.
The short version:
- An LLM router sent queries to the right representation 100% / 90% / 100% of the time. The keyword router managed 95% / 0% / 0%, blind to implicit intent.
- Real HippoRAG lifted multi-hop retrieval recall 0.65 → 0.99.
- Routed context matched "dump everything" on judged answers at ~⅓ the tokens, and clearly beat naive retrieval.
- On a tiny temporal haystack, the graph-temporal engine lost to plain retrieval. We publish that too.
The core idea: knowledge representation is a planning decision
The industry reflex to a hard retrieval problem is a bigger model or a bigger context window: both cost money and latency. Context Runtime v4 takes a different route, classify → constrain → learn: classify the query into a knowledge representation, constrain retrieval to that engine, then let an online bandit learn the best method per context.
┌─────────────┐
│ User query │
└──────┬──────┘
▼
┌────────────────┐
│ Classify the │
│ representation │
└──┬─────┬─────┬─┘
single │ │ │ history
fact ▼ ▼multi-hop
┌─────────┐ ┌────────┐ ┌───────────┐
│Document │ │ Graph │ │ Temporal │
│BM25/hyb │ │HippoRAG│ │ Graphiti │
└────┬────┘ └───┬────┘ └─────┬─────┘
└──────────┼────────────┘
▼
┌─────────────────┐
│ Bandit learns │
│ best method/ctx │
└────────┬────────┘
▼
┌─────────────┐
│ Answer model│
└─────────────┘
To test it honestly, the "right" answer for the router had to come from the dataset's design, independent of the routing model.
Public datasets: one per regime
| Regime | Dataset | Source (public) | Why it's here |
|---|---|---|---|
| Low / no graph | PopQA | akariasai/PopQA | Single-fact lookups: document should win |
| Multi-hop graph | MuSiQue | dgslibisey/MuSiQue | 2-hop over 20 paragraphs: graph's turf |
| Temporal conversation | LongMemEval (oracle) | xiaowu0162/longmemeval | Timestamped chat sessions: bi-temporal's turf |
Exact models: all local, all NVFP4, one 48 GB GPU
| Role | Model | License |
|---|---|---|
| Router · extractor · answerer | Qwen3.6-27B-NVFP4 (prithivMLmods/Qwen3.6-27B-NVFP4) | Apache-2.0 |
| Answerer | Mistral-Small-3.2-24B-Instruct-2506-NVFP4 | Apache-2.0 |
| Answerer | Gemma 26B-A4B (NVFP4) | Apache-2.0 |
| Answerer | NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 | NVIDIA Open Model License |
| Embeddings | BAAI/bge-large-en-v1.5 (1024-dim, CPU) | MIT |
| Judge | grok-4.5 (xAI), 578 rows judged for $0.51 | n/a |
Every answerer is 24–30B, deliberately no bigger model in the set, so this is a clean test of better context vs. more context at equal model size.
Real engines, real repos
| Layer | Repository |
|---|---|
| Orchestration | github.com/redevops-io/context-runtime · github.com/redevops-io/redevops-rag |
| Graph engine | github.com/OSU-NLP-Group/HippoRAG: OpenIE graph + Personalized PageRank |
| Temporal engine | github.com/getzep/graphiti: Neo4j bi-temporal knowledge graph |
Results
There was no universal winner, which is exactly the point.
1) Intelligent routing is the whole ballgame
Routing accuracy (→ correct representation) keyword ░ LLM █
PopQA → document keyword ███████████████████░ 0.95 LLM ████████████████████ 1.00
MuSiQue → graph keyword ░░░░░░░░░░░░░░░░░░░░ 0.00 LLM ██████████████████░░ 0.90
LongMem → temporal keyword ░░░░░░░░░░░░░░░░░░░░ 0.00 LLM ████████████████████ 1.00
The shipped keyword router collapses to 0.00 on the two hard regimes: "Who is the spouse of the Green performer?" has no "multi-hop" keyword, so it defaults to document and the graph engine is never reached. The LLM router nails all three.
The strongest test is one shuffled stream, so the router must decide per query from the query alone: low-graph 100% → document, multi-hop 93% → graph, temporal 100% → temporal.
2) On multi-hop, the graph engine earns its keep
MuSiQue retrieval recall (bootstrap 95% CIs)
fixed graph (HippoRAG) ███████████████████▓ 0.988 [0.963, 1.000]
CR (routed + learned) ██████████████████▌ 0.925 [0.863, 0.988]
fixed document █████████████ 0.650 [0.550, 0.738]
Δ(graph − document) = +0.338, 95% CI [0.250, 0.438], robustly significant. And you can watch the bandit learn it: cumulative recall climbed 0.750 → 0.850 → 0.900 → 0.925 as the chosen arm shifted toward graph.
3) Better context ≈ more context: at a third of the cost
Judged answer accuracy (grok-4.5), averaged across all four models:
MuSiQue answer accuracy
CR-routed context ████████████████████ 0.49 ← ~⅓ the tokens of full-dump
full-dump (all 20) ██████████████████▌ 0.46
naive (single-hop) ███████████▌ 0.29
- CR vs full-dump: Δ = +0.037, 95% CI [−0.025, +0.106] → a statistical tie. Matching "dump everything" while using ~⅓ the tokens is the win.
- CR vs naive: Δ = +0.206, 95% CI [+0.144, +0.269] → CR significantly wins.
- CR ≥ full-dump on every individual model.
PopQA ties across all methods (~0.97), precisely why routing to the cheapest representation there is an efficiency win.
4) The honest negative: temporal on a small haystack
We don't cherry-pick. On LongMemEval, routing was perfect (100% → temporal), but the temporal engine lost: Graphiti session-recall 0.731 vs document 1.000; judged answers full-dump 0.23 ≫ CR 0.03.
Why: the oracle split keeps only ~3 evidence sessions per question, a haystack so small that hybrid search finds every relevant one, and Graphiti's LLM-extracted facts are lossy versus the raw turns. Graphiti's real advantage (point-in-time reasoning over a large, revised history) needs the full haystack (hundreds of distractor sessions) to show. On this dataset, the correct representation is document. (Robustness check: doubling the efficiency penalty didn't change PopQA's route.)
The engineering takeaway: ship the LLM intent head
The benchmark's clearest signal: the v4 architecture is right, but its shipped keyword head scores 0.00 on implicit intent. The fix is a hybrid, confidence-gated analyzer, keyword-first (free on explicit cues), LLM only when the cheap head is unsure:
Query
│
▼
┌──────────────────┐ confident non-document rep
│ Keyword analyzer │ ───────────────────────────► use it (free)
│ (free) │
└────────┬─────────┘
│ document default / low confidence
▼
┌──────────────┐ agrees / unreachable
│ LLM head │ ──────────────────────────► keep keyword verdict
└──────┬───────┘
│ override
▼
new representation + re-align model tier
Implicit multi-hop → 1 LLM call → graph; explicit temporal cue → 0 LLM calls; LLM down → graceful keyword fallback. It carries a cost-ledger so the per-query routing overhead is measurable: the number you weigh against a model's built-in context handling.
Why it matters
This validated a core idea behind Context Runtime v4: knowledge representation is a planning decision, the choice of retrieval strategy. That now extends into Mission Runtime, which plans over knowledge representations, capabilities, models, applications, humans, and execution strategies, so enterprise AI will be differentiated by runtime architecture.
So before reaching for a bigger model or context window, ask whether you're even retrieving the right kind of knowledge, and let the honest failure mark exactly where the frontier still is.
Reproducibility: reward = recall − λ·cost (λ=0.15), seed=13, bandit ε=0.15; all NVFP4 on one 48 GB GPU. Engines: context-runtime, redevops-rag, OSU-NLP-Group/HippoRAG, getzep/graphiti. Datasets: akariasai/PopQA, dgslibisey/MuSiQue, xiaowu0162/longmemeval.
Data sources and licensing: the benchmarks use public datasets under their own licenses - PopQA (MIT), MuSiQue (CC-BY-4.0), and LongMemEval (MIT). Loaders download the data on demand; we do not redistribute it.
Built on open source. Every number is real, including the one that didn't go our way.
#ContextEngineering #RAG #KnowledgeGraphs #LLM #LLMOps #AIEngineering #MachineLearning #HippoRAG #Graphiti #Neo4j #RetrievalAugmentedGeneration #OpenSource #NVFP4 #GenAI #MLOps #AIInfrastructure