An AI answer is a sequence of routing decisions, not a fixed pipeline. Context Runtime makes each one - which representation, which retriever, which reasoning strategy, and how to compose and verify - a cost-based, measured choice. The benchmarks mirror that stack, top to bottom.
Which knowledge representation should answer this at all - analytical, document, graph, temporal, code or multimodal - and which method inside it? The decision above retrieval and reasoning. RepresentationBench: 300 curated questions through the real planner.
Which retriever within the representation? Knowledge-aware routing, reasoning-intensive retrieval (DIVER), the encoder A/B, and five-engine routing on real KYC data (BM25 · dense · HippoRAG · SQL · property graph).
Which reasoning strategy, and how much effort? A learned per-class decision: a thinking pass over full context lifts temporal from 0.79 to 1.00 - and, honestly, hurts multi-hop. Effort is routed, not global.
Compose several representations into one verified decision: resolve identity, cross-check, verify, and abstain or recommend. RepresentationBench-KYC + a harder matching-robustness half + Retail over real Walmart demand - Go/No-Go on real GLEIF + OpenSanctions data.
The decisions are orthogonal and they compose. A better retriever cannot fix the wrong representation; a better reasoner cannot answer from evidence it never received; and no single engine makes an onboarding decision. Optimizing each - and composing them - is what the runtime is for.
The layer behind the stack is specified in the v7 whitepaper - the AI query planner, choosing knowledge representations the way a database chooses access paths.
Planning decides which move; the Decision Runtime turns repeated decisions and their outcomes into an improving decision system. To isolate what the Runtime adds — not the model — we froze the base model (Qwen3.8-27B) and, arm by arm, added one Runtime layer at a time — verification, context, planning, bounded tools, discovery, persistent lessons — scoring every decision with a frozen Stockfish depth-16 oracle on 726 held-out chess positions. The full stack cuts the blunder rate 0.408 → 0.197 (−52%), same model throughout — and with the weights held fixed the held-out blunder rate falls 0.390 → 0.293 as the Runtime accumulates 13 learned lessons over 400 training decisions. No fine-tuning — the improvement is the Runtime's, not the model's. Chess is the controlled lab before auctions and revenue decisions.
The rAIlsApps benchmark now publishes a read-only evidence bundle: deterministic cross-app decision episodes, independent oracle verdicts, replay digests, behavior-policy pins, and learning eligibility. The first Lane A artifact has 2 governed episodes: one correct HOLD on an open dispute and one verified collection when no dispute exists. It is intentionally precomputed — visitors inspect evidence, not trigger live writes.
The mission-planning benchmark is the GRC third-party-risk decision. RepresentationBench-KYC — vendor onboarding over the real GLEIF + OpenSanctions graph — scores the composed Go/No-Go at 100% against a naive single-node screen's 36.7% (which onboards 45 of 50 sanctioned-linked entities, because the sanction sits on an owner upstream), and 99.5% on real aliases and namesakes.
grc.redevops.io — the vendor-onboarding journey walked end to end · the benchmark →
Planning decides what to do; the Governance Plane decides what a fleet of agents may not do — enforced over the whole trajectory, not one call. Measured on 460 real recorded GPT‑4o agent trajectories (τ‑bench retail, MIT), scored against the domain's shipped policy: it catches 38 multi‑event violations a per‑action gate catches 0 of, with zero cross‑order false alarms — and reports detection separately from enforcement (6 DENY where a gate exists, 12 NO_OVERRIDE where none does).
gov.redevops.io — step a real session call by call · the benchmark →
Evidence · controlled · internally reproduced 27 Oct 2026: 38 of 460 recorded trajectories carry an at-most-once violation (41 findings), precision and recall 1.00 on every rule; a per-action check catches 0 of the 38. The payment rule has no positive cases in this set.
A fleet of devices is the forcing function for the whole runtime — discovery at scale, reference-first context, cross-device governance. Measured on Backblaze Drive Stats — a real fleet, 5,061 drives · 1,061 real failures — an inspectable discovery rule on the telemetry trajectory clusters with trained RandomForest / XGBoost / GradientBoosting on the same signals — so the detector is an interchangeable provider, and Discovery's real job is optimal routing: run the cheap device-local capability first, escalate only the ambiguous cases. Adaptive routing matches an always-on model's precision and recall while running the specialist on just 7% of drives (14–42× fewer specialist inferences). The same routing generalizes to data-quality discovery — one Discovery call establishing whether the evidence can be trusted before another uses it — where gating corrupted telemetry lifts failure precision 0.300 → 0.850. Plus reference-first context (−95% bytes), fleet governance (96 quarantines → one review), and the physical ceiling (19% of failures show no SMART warning). Out-of-sample validated on a different quarter (Q1 2025): the frozen thresholds and models hold, often slightly better.
Evidence · controlled · internally reproduced 27 Oct 2026: false alarms 229 to 97 (57.6% fewer, paired p effectively 0) with recall 0.807 to 0.756 (paired difference -5.1 points over 54 drives); the adaptive-routing frontier is unchanged (0.852 / 0.723 at 0.07 times the specialist calls); data-quality precision 0.892 to 0.300 to 0.850.
The same Discovery-as-data-quality result in a second, agentic domain: an AI phone assistant modelled as composed Discovery stages (telephony → transcript → intent/slots → EHR/PM → retrieval → PHI/policy → action → TTS), each producing a trust verdict before acting. On a synthetic healthcare corpus (never real PHI), adaptive DQ discovery prevents 1,233 of 1,235 unsafe actions — wrong-patient lookups, unauthorized PHI disclosure, double bookings — at the specialist on just 13% of turns, and holds out-of-sample (1,825 of 1,827) on a shifted population — where it also honestly surfaces a small fairness disparity (0.000 → 0.034) that in-sample tuning hid.
Evidence · synthetic · internally reproduced 27 Oct 2026: 1,235 unsafe-if-ungated actions, 2 executed; out of sample 1,827 and 2; fairness disparity 0.034; regression suite 24/24; out of sample, 25 of 1,573 clean turns were gated unnecessarily.
The private enterprise plugins are measured on open data with the same honesty — misses reported as plainly as hits. First up, the epistemic first gate: a deterministic, zero-inference screen for prompt injection, scored on deepset/prompt-injections (CC-BY-4.0), patterns tuned on train and reported on the held-out test split. The enterprise corpus lifts recall 5× over the open baseline (0.05 → 0.25) at perfect precision and zero false positives — an explainable, replayable gate that never blocks a legitimate request and runs before a model classifier, not instead of one. Recall is honestly bounded (English-technique patterns vs a multilingual corpus).
Evidence · controlled · internally reproduced 27 Oct 2026: held-out recall 3/60 (open validators) to 15/60 (with the enterprise corpus), 0 false positives, paired p = 0.0005 (12 better, 0 worse).
What exactly does the Enterprise runtime add over the open (AGPL) one? We answer it with a falsifiable rule — Enterprise may add enforcement, evidence and verification; it must not silently change the open execution result unless a declared policy intervenes — and classify every differential as PARITY, HARDENED_DELTA, or BUG. Orchestration and safe concurrency come back PARITY (same state, nodes, scheduling); a governed mission is HARDENED_DELTA — authorized keeps the same execution and adds 5 hash-chained audit entries + verification, unauthorized runs 0 nodes because deny-by-default blocks it. The hardening is additive, not a fork of the execution semantics.
Does a runtime beneath your agents beat a plain LangChain / LangGraph flow? A 12-family benchmark — same workloads, same local Qwen3.8-27B, correctness beside cost — maps where it does and where it doesn't. ReDevOps wins where execution itself is the problem: replay recovers with 0 duplicate side effects (vs a re-run's 20/20 double-fires), context selection holds 100% accuracy on ~388× fewer tokens past the window, incremental redoes 6.1× less of the graph, and it is 2–2.3× lighter than LangGraph on raw orchestration. For pure fan-out with no shared state, a plain executor is the simpler choice — reported plainly. And safe concurrency (parallelize what's safe, serialize what must be, with an auditable reason) is conformance-tested across Python, Go and Kotlin — the Go runtime going 1.0× serial → 31× parallel at 32 tools.
// data sources & licensing - these datasets are used only to benchmark the runtime itself (not redistributed, not shipped in the product; customers apply Context Runtime to their own data). Each is used under its own license: GLEIF CC0 · OpenSanctions CC-BY-NC (non-commercial research use) · M5/Walmart Kaggle terms (non-commercial) · PopQA MIT · MuSiQue CC-BY-4.0 · LongMemEval MIT · TempReason CC-BY-SA-3.0 · τ-bench MIT (governance) · Backblaze Drive Stats MIT (fleet) · Voice: a synthetic healthcare corpus generated for the benchmark (no real PHI) · deepset/prompt-injections CC-BY-4.0 (enterprise). Per-benchmark details on each page.
// reproducible · deterministic where noted · github.com/redevops-io/context-runtime