DataOpsBench: Can an AI System Operate a Production Data Platform It Can't Fit in a Prompt?
The interesting failures in enterprise data work happen at scale, on the production version.
We showed that a first-class merge() operator becomes required once an integration exceeds a single model's context window: a feasibility boundary. That was mechanism; the product question:
Can an AI system operate a production data platform (diagnose incidents, trace lineage, reconcile totals, detect cross-source conflicts, repair pipelines, and preserve governance) when the relevant code, metadata, and history no longer fit in one model context?
So we built DataOpsBench, the DataOps analogue of TPC-C: can a system continuously operate the platform while preserving correctness, lineage, evidence, and governance? Its architecture is the v9 architecture whitepaper.
The setup: one variable, deterministic gates first
The platform is synthetic but realistic: "Northstar" plus an acquired source "Meridian" that disagrees on purpose (different keys, currency, definitions). Every scenario ships a seeded defect, hidden ground truth, and a deterministic verifier, so correctness comes from real checks (row counts, reconciliation totals, executable views, parsed lineage edges) before any subjective judgment.
Two arms differ only in orchestration, with identical tools:
- Direct: a capable model over the raw platform.
- Runtime: the same models through rAIls: bounded
merge(), structured ledgers carrying lineage/entity/quality facts, deterministic operators, and governed missions.
Note: the runtime often doesn't put the model in the loop at all: for structured work it plans deterministic operators over bounded context, the point; we're explicit below where the model is used.
The floor: when it fits, they tie
On small, in-context repairs (a duplicate-after-retry, a source-vs-finance reconciliation, a lineage break after a rename) both arms pass 3/3. A direct model resolves anything that fits its window; the runtime adds nothing there. That sanity floor stops the next results looking cherry-picked.
Where it stops fitting
Consolidation at scale. Consolidate the whole lineage and reconcile a cross-source total. The direct arm crams everything into one prompt and overflows the window; the runtime works through its deterministic ledger operators, exact at scale 20 → 2000 pipelines (≈9K → 908K tokens, 28× the window), zero model calls, no overflow.
Repair localization. A table is renamed and broken references hide across the platform; the task is to find all of them. The direct arm finds every one while the platform fits the window (recall 1.0, full competence), then collapses to 0 once it can't; the runtime's lineage scan finds them all at any scale.
| task | direct arm | runtime |
|---|---|---|
| Consolidation (lineage + cross-source total) | overflow past the window | exact to 908K tokens · 0 model calls |
| Repair localization (recall) | 1.0 in-window → 0.0 past it | 1.0 at every scale · 0 model calls |
Same as the merge test: the wall is the context window: change nothing but platform size and it lands in the same place.
The acquisition problem: detecting what the merge hides
Meridian was introduced to disagree on purpose. So the sharper M&A question, after consolidation above, is "do you surface every reconciliation conflict the merge introduces?" The same logical metric (a KPI, a revenue line) reported by Northstar and by the acquired Meridian with different values, buried among thousands of clean ones.
This exposes a second way a direct approach fails: one with nothing to do with the context window. Three approaches to the same merged platform:
- Direct model: read the whole platform, list the disagreements. Correct while it fits (recall 1.0), then overflows once the platform exceeds the window, the familiar wall.
- Naive consolidation: union the facts under a token budget, keyed on the value. This fits at any scale and never overflows, and silently reconciles every disagreement away (recall 0.0, even at trivial scale). Two facts that differ are just two facts; the conflict is never surfaced.
- Governed
merge(): union keyed on the logical metric, so Northstar's value and Meridian's value for the same metric collide as a conflict. Every one, at any scale, precision 1.0.
| approach | fits the platform? | conflicts detected |
|---|---|---|
| Direct model | no: overflows past the window | 1.0 in-window → 0.0 past it |
| Naive consolidation (value-keyed) | yes, at any scale | 0.0: silently reconciled away |
Governed merge() (metric-keyed) | yes | 1.0 · precision 1.0 at every scale |
The naive row is the point. It doesn't overflow (it scales cleanly to any size) and it's wrong the whole time. "Make it fit" is not the same as "get it right": a system can consolidate a merged platform without a hiccup and quietly lose every conflict that made the merge risky in the first place. Only the governed operator (same bounded working sets, only the union key differs) is right and scale-free, at zero model calls. Distributed on Spark it holds across 263 executor tasks at 2,000 pipelines, the conflict surfacing exactly where the two sources meet.
Where the model does earn its keep: grounded reports
Incident reports, root-cause explanations and docs are a model's job, and here both arms use the model. The difference is what it's given: the runtime grounds the report in verified evidence (the exact reconciled total its operators computed), the direct arm in chat.
- Deterministic groundedness gate: direct 0.75 vs. runtime 1.00: the single miss is that the direct report can't reliably state the verified total.
- A blinded, pairwise judge from a different model family (a local DeepSeek, since the arms are Qwen), report order randomized to cancel position bias: runtime 4, direct 0, for grounded reasons, every verdict citing the correct figure from ground truth.
The judge is secondary to the deterministic gate and agrees with it: the runtime's edge is feeding the model verified facts.
Proactive detection and auditable execution
Two more capabilities, through the actual Discovery and Mission runtimes:
- Discovery (no prompt): scanning lineage state, it proactively proposes a repair for every broken reference: recall and precision 1.0 at up to 2,000 pipelines, before anyone asks.
- Governed missions: each proposal compiles into a repair mission whose side effects run exactly once on the control plane, with a hash-chained event ledger that self-verifies: at 2,000 pipelines and 1,002 events, the ledger checks out.
The honest read
Three consistent findings, more nuanced than "our AI is smarter": (1) the runtime's first advantage: deterministic operators over bounded working sets are exact, scale-free and cheaper (zero model calls), where cramming the platform into a prompt overflows; (2) when the model is the right tool (explanations, reports), the advantage is grounding it in verified evidence; (3) a direct approach fails two ways: it overflows the window, or (where it does fit) it silently gets the wrong answer, reconciling away the cross-source conflicts. The runtime is the only thing right on both: feasibility and correctness. That two-axis gap is the regime enterprises operate in.
The honest limits
DataOpsBench is a mechanism and capability test outside leaderboard use. The platform is synthetic (reproducible ground truth, no customer data), the arms use one model family, several runtime-side families are deterministic, and the judge touches only the report family. It measures that and how the runtime crosses the context boundary: model- and framework-agnostic, scoped to this boundary.
Try it: DataOpsBench is open source
The benchmark itself is open (Apache-2.0): github.com/redevops-io/DataOpsBench. Scenarios, deterministic gates, and reference solvers are all there.
python3 dataopsbench.py validate # RESULT 5/5 — the gates discriminate defect from fix; no keys, no model
python3 run_arm.py s20 20 60 120 # cross-source consolidation under overload: direct vs. bounded
python3 run_arm.py s14l 20 60 120 # lineage-repair localization at scale
python3 run_arm.py s21 20 100 300 600 # cross-source reconciliation-conflict: direct vs. naive vs. merge()
Your own agent (LangGraph, Codex, Cursor, a custom runtime) plugs into the same arm interface, scored by the same verifiers. A benchmark others can run (and win) on is credible where a self-scored one never is; the runtime behind the numbers is just our solver on this workload.
The discipline we apply everywhere: measure whether a system can keep operating a platform once that platform outgrows a single context window, and let the evidence decide.
#DataOps #DataEngineering #AIEngineering #LLMOps #ContextRuntime #MissionRuntime #Lineage #DataQuality #Reconciliation #Governance #Benchmark #GenAI #Agents #AIInfrastructure