← redevops.io  /  Benchmarks  /  Mission ↑ Representation
Mission planning · compose · verify · abstain · recommend

The Go / No-Go decision

The top of the stack. Representation planning chooses one access path; mission planning composes several into a governed decision - resolve identity, gather evidence across representations, verify, and abstain or recommend. The test is deliberately not "can the graph database find owners" - everyone knows it can. It is: can the planner compose that evidence into the right decision, for the right reason?

the planning stack:   representation → retrieval → reasoning → mission  |  you are here: mission

RepresentationBench-KYC

150 onboarding scenarios over the real GLEIF + OpenSanctions graph (182,445 entities in Neo4j), labelled as facts about the data: NO-GO (a sanctioned entity anywhere in the ownership chain), GO (clean chain), ABSTAIN (the name matches more than one entity - identity unresolvable, escalate). The decider composes: resolve identity → traverse ownership → screen every owner for sanctions → verify and recommend.

150 scenarios · 50 Go / 50 No-Go / 50 AbstainComposed decision
(the planner)
Naive single-node
screen (baseline)
Decision accuracyGo / No-Go / Abstain vs ground truth100%36.7%
Sanctioned entities onboardeda No-Go waved through as Go - the dangerous error045 of 50
Abstains on ambiguous identityescalates instead of guessing the wrong entity50/500/50
Explanation correctNo-Go cites the exact sanctioned node100%-
Decision latencyfull compose + verify + decide0.6 ms-
composed 100%   naive 36.7%   the gap is the value of composition

The sanction is upstream 45 of the 50 No-Go applicants are clean at the name - the sanctioned entity is an owner, one to eight hops up. A single-representation sanctions lookup sees a clean applicant and onboards it. Only composing the ownership traversal with the screen catches it.

Abstention is a first-class outcome All 50 ambiguous-identity cases are escalated, not guessed. A KYC system that auto-decides on the wrong "Acme" is worse than one that asks. The composed decider abstains; the baseline never does.

Right decision, right reason Every No-Go cites the exact sanctioned node, and the full chain is preserved as replayable evidence. The recommendation is auditable months later, not a black box.

What it is not Not a test of Neo4j (assumed to work) and not a fuzzy human judgement (labels are deterministic facts). It measures the runtime's composition and governance of the decision.

What is real, what is derived. The entity names and the 168,818 ownership edges are real GLEIF golden-copy records. The sanctioned flag is derived: a GLEIF entity is flagged when its name matches an real OpenSanctions target name by exact, case-insensitive string match (3,765 of 182,445 entities). So a flag is a real match, not an injected label - but it is an exact-name match, which is deliberately the assumption the matching-robustness half below stress-tests. One consequence: a name collision can flag a legitimate entity (e.g. a "Standard Chartered" subsidiary), which is harmless as a graph-traversal fixture but is why the composition, not the flag, is the thing under test here.
Scenario composition. 150 onboarding scenarios, evenly split 50 GO / 50 NO-GO / 50 ABSTAIN, curated by querying the graph: NO-GO = a clean-named applicant with a flagged owner 1-8 hops up its real ownership chain; GO = a fully clean chain; ABSTAIN = a name resolving to more than one entity. Full curator + labels: CR-enterprise/benchmarks/representation-bench-kyc/curate_scenarios.py.

// 150 scenarios over real GLEIF (182,445 entities) + OpenSanctions in Neo4j · composed decision vs a naive single-node screen · decision accuracy · abstention · explanation · latency · reproduce: CR-enterprise/benchmarks/representation-bench-kyc/

The harder half: matching robustness

Fair criticism of the benchmark above: the applicant already arrives as an exact graph node, and the labels are deterministic, so the composed pipeline scores 100% by construction. This second benchmark removes that crutch and tests where the composed decision can genuinely be wrong: matching the applicant to the sanctions list when the name does not arrive as an exact string. 200 scenarios built from real OpenSanctions names, aliases and countries and real GLEIF names - exact hits, real aliases (LLC System Industrial Group is listed as LLC Sistema PG), typo near-matches, and coincidental namesakes in a different jurisdiction.

200 scenarios · real aliases + namesakesAccuracyDangerous
misses
False
positives
Exact-name screenmembership in the sanctions list - the baseline60.0%800
Fuzzy name-only screenrapidfuzz over names + aliases, no other signal62.5%040
Composed: fuzzy × jurisdictionflag on the name, resolve identity by country99.5%00
composed 99.5%   exact 60% · 80 misses   fuzzy 62.5% · 40 false-pos

Exact matching is dangerous 60% looks fine until you see the errors: 80 sanctioned-derived names waved through - every real alias and every typo variant. High apparent precision, zero recall on the cases that matter.

Fuzzy matching over-corrects It catches 100% of the real aliases but over-blocks 40 clean namesakes - it cannot tell a real hit from a coincidental name. In production that is alert fatigue and wrongful de-risking.

Composition resolves the trap Flag on the name, then resolve identity by jurisdiction: a fuzzy hit in a different country is a different entity. Zero dangerous misses and zero false positives.

Not 100% by construction The single error is a near-match classified No-Go instead of Abstain - over-strict, not dangerous. The composed screen can be wrong, which is the point: 99.5% is a real robustness number, honestly earned.

Scenario composition. 200 scenarios, 40 each of five types. exact and alias are real OpenSanctions primary names and real aliases (unchanged). near is derived - a real sanctioned name perturbed by dropping its last token or swapping two characters (a typo variant). namesake is a real clean GLEIF name selected for a moderate fuzzy similarity (WRatio 70-88) to some sanctioned name and a different jurisdiction - the coincidence trap. clean is a real GLEIF name with low similarity to any target. The three screeners: exact (list membership); fuzzy (rapidfuzz WRatio, ≥92 → No-Go, 80-92 → Abstain); composed (fuzzy to flag, then clear a hit whose jurisdiction differs). Full construction + thresholds: representation-bench-kyc/MATCHING.md.

// 200 scenarios · real OpenSanctions names + aliases + countries vs real GLEIF names · three screeners of increasing composition · dangerous misses (a sanctioned name screened Go) + false positives (a clean name blocked) · reproduce: CR-enterprise/benchmarks/representation-bench-kyc/matching_bench.py

RepresentationBench-Retail

The same architecture, a different domain: should we reorder this SKU? composed from camera (shelf reality), inventory (system on-hand), and POS (sales velocity). 204 scenarios over the real M5 (Walmart) data - and here we use the signals M5 actually ships: real price elasticity (measured per SKU from real sell_prices) and real event uplift (from the calendar), plus a camera that is allowed to lie. That is what makes the composed decision able to fail.

204 scenarios · real M5 calendar + pricesAccuracyStockouts
missed
False
reorders
Naive inventory-onlyreorder if system on-hand is low31.4%840
Composed: trust camera + trailing velocitythe first benchmark's decider, on the harder set54.9%568
Composed+: corroborate + calendar/pricereconcile camera × inventory × POS; real demand model96.1%08
composed+ 96.1%   trust-camera 54.9%   inventory-only 31.4%

The naive 100% was an artifact Add the signals M5 actually contains and a "trust the camera + trailing velocity" decider drops from 100% to 55%: it is velocity-blind (misses every promotion and event demand spike) and over-trusts a noisy camera. The earlier benchmark simply never tested these cases.

Composition recovers it, honestly Corroborating camera against POS kills the false reorders from a noisy camera (POS still selling ⇒ disbelieve the empty shelf, escalate), and the real calendar/price demand model catches the promo and event stockouts. Zero real stockouts missed.

The blind spot is reported A camera glitch that coincides with a real demand lull is observationally identical to a true stockout - no reconciliation over three signals can separate them. composed+ resolves toward restock (safe for real stockouts) and over-orders on all 8 traps. 96.1%, not hidden.

The phantom stockout is the retail analogue of KYC's "the sanction is on an owner upstream": inventory says stocked, the shelf is empty, and the answer is not in the representation you queried. Story Runtime is intentionally deferred - continuity metrics are not mature enough to score fairly yet.

What is real, what is modelled. Real, per SKU from M5: sales velocity (trailing 28-day mean of true daily units), price elasticity (the demand ratio at the SKU's low vs high real sell_price - only SKUs with a genuine ≥1.4× response become PROMO cases), and event uplift (the demand ratio on the SKU's best event-type days, ≥1.4× to qualify for EVENT). Modelled/injected: on-hand inventory and the shelf-camera reading, which is deliberately allowed to disagree. Scenario composition. 204 scenarios: 28 each of HOLD, RESTOCK, PHANTOM, PROMO, EVENT, NOISY_VISION and conflict-ABSTAIN, plus 8 NOISE_TRAP (the adversarial camera-glitch-meets-demand-lull case that defeats corroboration). Full construction: retail-mission/DATASET.md + HARD.md.

// 204 scenarios · velocity, price elasticity and event uplift measured from real M5 Walmart series; on-hand + a fallible camera per the disclosed model · three deciders of increasing composition · reproduce: CR-enterprise/benchmarks/retail-mission/retail_hard.py

The whole stack, one trace

The four benchmarks measure each stage in isolation. End to end, one real onboarding question flows through all four in a single sub-10 ms run - each stage's output is the next stage's input, nothing mocked: the planner is real, the graph is real Neo4j.

applicant SCBMB TRUSTEE BERHAD (MY)  -  a clean name; a sanctions screen returns nothing

1 · representation  question → bucket=multi_hop → representation=graph, method=cypher  ~0.5 ms
2 · retrieval      cypher traversal: 1 identity resolved (LEI), 7 ownership hops  ~7 ms
3 · reasoning     screened all 8 chain nodes + the name; 2 sanctioned owners; verify required  ~2 ms
4 · mission       NO-GO, citing STANDARD CHARTERED PLC (GB) at hop 7  ~0.01 ms

// one workload, four planning stages, one trace · real planner (Context Runtime) + real Neo4j (GLEIF + OpenSanctions) · reproduce: CR-enterprise/benchmarks/stack-e2e/stack_trace.py

The planning stack, measured: representation → retrieval → reasoning → mission. The layer behind it: v7 - the AI query planner.
Data sources & licensing
These datasets are used only to develop and benchmark the Context Runtime software itself - they are not redistributed, not embedded in the product, and not part of any customer deployment. Context Runtime customers apply it to their own data under their own licenses. Each dataset is used under its own license; loaders download it on demand.
  • GLEIF golden-copy (LEI + relationship records) - CC0 1.0 (public domain).
  • OpenSanctions (default targets) - CC-BY-NC 4.0; used here for non-commercial research only - commercial or compliance deployment requires a separate OpenSanctions data license.
  • M5 Forecasting-Accuracy (Walmart), via the denephew/M5_Forecasting mirror - Kaggle competition terms (Makridakis et al., 2022); non-commercial research use.
redevops.io is a member of the NVIDIA Inception program. © 2026 NVIDIA, the NVIDIA logo and NVIDIA Inception are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries.