We Benchmarked Formal Multi-Agent Orchestration Against a Good Supervisor. It Tied. Until It Couldn't Run.
Measure the mechanism against the simple baseline before you build it, and let the boundary decide ahead of aesthetics.
Modern multi-agent frameworks sell formal machinery: rigid isolation contracts, deep agent hierarchies, generic semantic merge operators, runtime-managed integration. It sounds more principled than a single model coordinating a few workers. "Sounds more principled" is not evidence.
So before committing any of it to a roadmap, we ran two controlled tests against one baseline: a strong supervisor that decomposes work, runs parallel workers, and integrates their output directly. The first asked whether formal orchestration is better. The second asked whether the baseline still works at all when the input outgrows a single model. The two answers, together, are what put one specific operator into the runtime. The full harness is open source: github.com/redevops-io/multiagent-orchestration-benchmark.
The pre-registered rule
Adopt a formal mechanism only if it holds gate-pass rate and wins on quality or on cost/latency. Ties default to the simpler system.
Pre-registering the decision rule matters: with enough metrics you can always find some axis where the fancier system looks better. Fix the rule before running anything.
Test 1: is formal orchestration better? Setup with one variable
The entire design holds everything constant except the orchestration layer, so any difference is attributable to that.
- Manager: the same model (GPT-5) in both arms. It decomposes, briefs, reviews, and integrates, but never writes feature code. A representative manager on purpose: a too-strong one could paper over orchestration lapses and hide the effect we want to measure.
- Workers: a fixed pool (Grok-4.5 + Kimi-k2.7-code), each a single-task agent in an isolated git worktree.
- Assignment: random, seeded, and paired: the identical subtask-to-model map in both arms, deliberately not skill-matched, so worker quality can't mask the integration difference.
- Topology: predeclared per task shape, identical in both arms.
- Judge: an independent model family that writes none of the code, scoring blinded and pairwise with randomized order. It only breaks ties among solutions that already pass the deterministic gates.
Arm A (emergent) sends natural-language briefs and the manager integrates the parts directly. Arm B (formal) sends exact-contract briefs, then a runtime integration mission runs a declared-output check, a one-shot recovery retry, and a compile-probe before handing that state to the manager. Semantic authority stays with the manager in both.
The tasks are three Kotlin shapes at different dependency structures (independent modules, a shared serialization contract, and a cross-cutting refactor through a Capability interface) plus two failure injections. Correctness is decided first by deterministic gates: a real compiler build, a hidden-test suite, and a static architecture scan. Only then does the blinded judge weigh in. (Each task is an executable specification, a Python driver plus a frozen target spec, the same principle behind why Python is the cleanest specification for code work: the hard part is pinning down behavior and contracts precisely, past syntax.)
An honesty checkpoint worth telling
Our first run looked like a win for the formal arm: 3/3 vs 2/3. It was a lie the harness told us. The manager's integration call had a token cap too low for a reasoning model; on one run GPT-5 spent the whole budget "thinking" and returned a zero-byte solution, which the gate scored as a failure for the emergent arm. It looked like formal orchestration recovering where emergent broke. It was a bug in our own scaffolding. We fixed it (escalate the budget on an empty completion; treat persistent-empty as a harness error, never an arm failure) and re-ran everything uniformly. Reporting that artifact as a finding would have been the easiest, and most wrong, thing to do.
The result: a tie on every axis
Across all 11 paired comparisons the two arms tied on correctness (11/11 each, both injections recovered by both), on cost, and on code quality (5–5–1, blinded); latency showed no consistent winner. By the pre-registered rule, ties default to the simpler system, so on these tasks, the evidence does not support adopting formal runtime-managed orchestration. A capable supervisor integrating work directly matched the formal pipeline with less machinery.
The practical read: keep the cheap plumbing (worktree isolation, provenance, budgets, scoped briefs): it's low-risk and useful regardless. Don't build generic merge operators, rigid isolation contracts, or deep hierarchies on the strength of "it feels more principled."
But this is a mechanism test, leaderboards aside, and deliberately small: the Kotlin tasks are toy-scale and both injections were recovered by both arms, so they may simply not stress the formal arm's hypothesized edge. If formal machinery has an advantage, it should show on work larger and more interdependent than this. That is exactly where we pointed the follow-up.
Test 2: does the baseline still work at all? The feasibility wall
At small scale the interesting axis is quality. At large scale it's feasibility. Same discipline, sharper question: consolidate a growing pile of real documents (Databricks' AI blog, 278 posts) into one non-redundant digest, using a fixed 32K-context model as the integrator, distributed on Spark.
- Direct merge, the supervisor move Test 1 vindicated: one model call over everything.
- Bounded hierarchical
merge(), a governed operator: chunk the input under a token budget, merge each group, recurse (module → subsystem → whole), so every individual call stays under the window.
The result is a wall, past the tie.
Direct merge's prompt climbs with the input (6.6K, 14K, 17.7K, 23.1K) until it crosses the window, at which point the model rejects the request outright (HTTP 400: "maximum context length is 32768") and never completes again. Bounded merge() completes every point from 9K to 165K tokens (an 18× range) with its peak per-call working set held flat at ~6.6–8.6K, because hierarchical depth (1 → 4 levels) absorbs the growth instead of the context window. Below the window the two arms are on par; merge() buys nothing until the context is actually exceeded, and above it, direct merge is impossible.
So the two tests are one rule: don't adopt a mechanism because it feels principled (at small scale, formal orchestration merely ties), and do adopt a specific operator when a measurement shows the simple baseline can't do the job at all (above the window, the supervisor can't run). We added merge() on exactly those terms: opt-in, planner-selected, and required only where a single model can't hold the integration context. In the runtime that's literal: the merge operator is optional on the single-node profile (direct integration is the default) and a required dependency on the Spark profile, where "input too large for one machine" is the founding premise.
From measurement to operator: what makes merge() more than compaction
A feasibility win invites a fair critique: maybe bounded merge() is just dressed-up chunk-and-summarize. If that were true, it wouldn't deserve a place in an operator algebra. So we made the distinction load-bearing: four properties, each now hardened and, as of this release, implemented at parity across all three runtime references (the Python executable spec, the Go systems port, and the Kotlin/Spark runtime):
- Verification gates propagation. A merge integrates upward only if it passes: every inherited contract token must survive, no established fact may regress, and no high-severity conflict may be left unresolved. A merge that silently drops a fact (the exact failure a summarizer hides) fails and does not propagate.
- Conflicts are semantic, past collisions. It separates an interface incompatibility (same symbol, different signature: blocks) from a benign identical duplicate, and detects cross-source contradictions (two sources reporting different values for the same key, the case that bites when you integrate an acquired system whose data disagrees with yours).
- The planner selects the topology. Not always hierarchical: it chooses direct / hierarchical / contract-first / sequential-verified from context size, coupling, and risk, feasibility-first, so direct is excluded above the window.
- It learns. Whether a merge passed verification, and at what cost, feeds back per context band, so the topology choice improves from experience instead of staying fixed.
Together with the structured ledgers that carry facts through compression, that's the line between a governed integration operator and context compaction: it verifies, detects real conflicts, plans its own strategy, learns, and refuses to propagate anything it can't stand behind. The same logic governs the rest of the operator algebra (plan · discover · retrieve · verify · promote · rollback · learn · explain), primitives the planner issues when a measurement says to, never blanket machinery. The full design is in the v9 architecture whitepaper.
The honest counterweight, now measured. Bounded merge()'s prose digest does lose fidelity as the hierarchy deepens (recursive summarization drops detail). But that's the naive failure mode, and it's exactly what the structured ledgers prevent. A third test ran naive bounded merge against governed merge() on the same corpus, same model, and same call count: the only difference being the governance layer. Across a depth ladder, the naive prose digest retained just 14–36% of the corpus entities and got worse with scale, while the governed ledger held ~98%, flat:
| Input | posts · levels | naive prose (Arm A) | governed ledger (Arm B) |
|---|---|---|---|
| 24K | 6 · 2 | 0.36 | 0.98 |
| 48K | 12 · 2 | 0.14 | 0.99 |
| 100K | 23 · 3 | 0.18 | 0.98 |
Same integrator, same 3 / 6 / 13 merges per side: the entire gap is the deterministic ledger union carrying the facts the recursive summary throws away. Conflict detection was a tie at this small scale (both arms flagged all five seeded contradictions, which sit in early posts). So we ran it deeper.
A larger run pushed the ladder to 100K → 350K tokens (23 → 74 posts) with 15 contradictions spread throughout the corpus, and it confirmed both edges:
| Input | posts · levels | naive coverage | governed coverage | naive conflict recall | governed conflict recall |
|---|---|---|---|---|---|
| 100K | 23 · 3 | 0.18 | 0.98 | 1.00 | 1.00 |
| 200K | 41 · 3 | 0.17 | 0.98 | 0.38 | 1.00 |
| 350K | 74 · 3 | 0.12 | 0.98 | 0.00 | 1.00 |
Coverage held ~0.98 flat to 74 posts while naive prose fell to 0.12, a gap that holds past small scale. And the conflict edge, a tie when the contradictions were few and early, emerged with depth: as the hierarchy deepens, naive recursive summarization buries the conflicting values, so the naive arm's recall degraded 1.00 → 0.38 → 0.00 (catching none of the 13 contradictions at 74 posts) while governed merge() caught all of them at every scale, at 1.0 precision (zero false positives). Naive conflict detection is a fragile heuristic that decays with scale; governed detection is a deterministic guarantee that doesn't. So feasibility was the first question these tests settled; coverage and trustworthy conflict-handling are the second, and the ledger answers both.
Reproduce it
Everything is open source: github.com/redevops-io/multiagent-orchestration-benchmark. python3 validate_gates.py re-checks the gates against reference solutions with no API keys and no model calls; harness/bench.py <shape> <seed> [injection] runs a paired trial with your own keys (read from the environment: the repo contains no credentials). Raw per-run results and per-pair judge verdicts are included.
The lesson is the discipline we apply everywhere, beyond "multi-agent is bad": don't standardize on a mechanism because it feels sophisticated: measure it against the simple baseline, and let the evidence decide. Test 1 said don't build the machinery on aesthetics. Test 2 said build this one operator, because the baseline physically can't run. Both came from the same rule.
#MultiAgent #AgentOrchestration #AIEngineering #LLMOps #Benchmark #Reproducibility #SoftwareEngineering #GenAI #Agents #MissionRuntime #ContextRuntime #LangGraph #CrewAI #AIInfrastructure