What a Runtime Buys You That LangChain and LangGraph Don't, and What It Doesn't
A 12-family benchmark of ReDevOps against LangChain, LangGraph, and a hand-rolled baseline, plus cross-language runtime conformance across Python, Go, and Kotlin, measuring correctness beside efficiency on real corpora with a local 27B model.
The question
If you are building agents, you can wire tools and prompts together directly (a few functions and a loop), reach for LangChain or LangGraph, or put a runtime underneath the agents that owns orchestration, context, governance, and replay. The pitch for the last option is that a runtime pays for itself on the things that actually break in production: a double charge after a crash, a model that silently drops half the requirements, an answer that collapses the moment the context window fills. The pitch against it is overhead and complexity you may not need.
We wanted numbers. So we built a benchmark that runs the same workloads through four implementations and reports correctness next to cost for each:
- minimal: a hand-rolled baseline (thread pool + direct calls). The "do you even need a framework?" arm.
- langchain: LCEL runnables (
RunnableParallel, retrievers, chains). - langgraph: a
StateGraphwith checkpointing. - redevops: rAIls (Mission Runtime + Context Runtime + RAG), driven only through its public SDK, never its internals.
Every arm sees the same model and the same retriever (we call this Track A fairness): shared BM25, identical top-K, identical prompt, one model endpoint. When we let a runtime use its native capability (context selection, hybrid retrieval, routing), we label it Track B and say so. The model throughout is Qwen3.8-27B-NVFP4 on vLLM, on a single 32 GB RTX PRO 4500, deliberately a local, mid-sized model, because that is where "what the model sees" matters most.
The datasets
We reused real corpora wherever they existed:
theme dataset
--------------------------------- --------------------------------------------------------
orchestration & concurrency synthetic deterministic DAGs (5/20/100 nodes) + fan-out
basic RAG LiveRAG anti-parametric corpus (obscure FineWeb facts)
context pollution & long context FinanceBench (150 QA / 84 filings) + a medical corpus;
needle-in-haystack from 10k to ~1M tokens
incremental recomputation Wikimedia revision history (sha256-pinned strategywiki)
replay / recovery a mission with a real side-effecting step
representation routing KYC over GLEIF beneficial-ownership (BM25/dense/graph/SQL)
deterministic-first (zoning) official zoning/geospatial data, false-permitted = 0 SLO
agentic voice / data quality ~24 adversarial scenarios (spoofed audio, injection, ...)
intent compilation (RAAAL) Quantify strategy prompts, silent-reduction defect class
governed multi-step mission an 8-step revenue pipeline with an approval-gated send
Grading is deterministic and gate-first: numeric match + claim coverage, decided before any LLM judgment, so results are reproducible.
The results, by what actually drives them
The point of the benchmark is to show where an architectural feature pays off and where it doesn't. Five runtime properties explain most of the benchmark results. A sixth result, cross-language conformance, tests whether those semantics survive outside the original Python implementation.
1. An event-sourced mission ledger → safe recovery and governed side effects
The Mission Runtime records every step as an immutable, causally-ordered event, and can seal a run into a self-verifying case bundle. That single design choice is what shows up in two families:
- Replay / recovery (Family 7). After a crash, replaying the sealed bundle recovers with 0 duplicate side effects across 20/20 recoveries, consistency- and integrity-verified every time. A framework that "recovers" by re-running the graph double-fires the side effect on every recovery: 20/20 double charges. LangGraph checkpointing is a partial middle ground (resume from a checkpoint, but the step after it still re-fires); LangChain has neither.
- Governed multi-step mission (Family 12). The 8-step revenue mission completes under approval (the send fires exactly once), and without approval it parks at the gate (
state=waiting_human, 0 sends), resumable, then replays with 0 duplicate sends. The plain LangChain/LangGraph pipeline completes the same steps but offers neither the native approval-park nor side-effect-free replay.
This is the clearest "runtime earns its keep" result: idempotent recovery and an approval gate that actually holds the consequential action are properties you get from the substrate.
2. A Context Runtime that chooses what the model sees → a small model that punches up
The most consequential wins are about context selection. On a local 27B model:
- Context pollution (Family 4). At 95% noise, selecting the right context scores +7 accuracy points on 93% fewer tokens than stuffing everything in. Cleaner beats more.
- Long-context resurrection (Family 5). Feeding the full document collapses accuracy from 100% → 0% once it exceeds the window. Selecting the relevant slice holds 100% accuracy on 84 tokens versus 32,612, roughly 388× fewer tokens, which is the difference between the local model being useless and being correct. This is the feature that makes a 27B viable past its context limit at all.
- Incremental recomputation (Family 6). When evidence changes, ReDevOps recomputes only the affected causal region: 16.5% of the graph, or 6.1× less work, while producing identical final state. Evidence identity and dependency tracking turn a source update into targeted invalidation.
LangChain and LangGraph have retrievers, but "retrieve then stuff" is not the same as a runtime that models context as a budgeted, explainable selection with an as_of view, and tracks which downstream state depends on which evidence. On the pollution, long-context, and incremental-recompute workloads, that distinction is the result.
3. Capability metadata + a scheduler → safe concurrency and routing
The runtime attaches typed metadata to each capability (resource/conflict keys, a concurrency mode, cost) and schedules against it:
- Safe concurrency (Family 2). Concurrency itself is commodity. The runtime feature is safe concurrency: deciding which work may overlap without weakening execution semantics. Independent work parallelizes; provider fan-out obeys local concurrency bounds; two writes to the same resource key serialize while different keys run together; approvals remain branch-local; a failed branch does not duplicate a sibling's effect; and governance containment cancels pending work while preserving completed effects. The scheduler records why work was held (for example,
"serialized: resource_key=k8s:cluster:prod at limit 1"), so parallelism is part of EXPLAIN. LangChain and LangGraph can parallelize independent nodes; the additional runtime claim is conflict-aware scheduling of consequential work.
- Representation routing (Family 8). Choosing the right retrieval representation per query (BM25 / dense / graph / SQL) routes correctly 92.7% of the time versus 16.7% for a fixed representation, at 0.11 ms per decision. The metadata makes the routing decision cheap and principled.
4. Deterministic-first execution + validator chains → safety SLOs a prompt can't guarantee
Where a wrong answer is a safety event, the runtime decides with deterministic gates before any LLM judgment, and routes to a specialist/validator only when needed:
- Zoning (Family 9). The learned policy commits 0 false-permits at 1.4× lower cost than always running the expensive path; a single-provider baseline violates the SLO.
- Agentic voice (Family 10). Adaptive gating lets through 2 unsafe turns versus 1,235 for the naive baseline, 99.8% of unsafe actions prevented, while escalating to the costly specialist on only 13% of turns.
- Intent compilation (Family 11). A danger-split score catches 8 SILENT_REDUCTION defects (a plan that quietly changes the requested result) that a naive accuracy benchmark would happily reward.
These are the results you cannot get by making the model "try harder": the guarantee comes from the gate.
5. The cost of all this, and where it isn't worth it
None of the above is free, and the benchmark says so:
- Orchestration overhead (Family 1). On a bare deterministic DAG with no model call, ReDevOps adds real per-node overhead, but it is 2–2.3× lighter than LangGraph (linear-100: 14.8 ms vs 26.1 ms), with all four frameworks 100% correct. The governed, event-sourced machinery costs less than LangGraph's state management.
- Raw fan-out (Family 2, the honest frontier). For independent I/O with no shared resources, approvals, governance, replay requirements, or provider limits, all you need is concurrency: a plain executor may be the simpler choice. ReDevOps earns its additional machinery when parallel work starts touching shared resources or consequential side effects, because concurrency then becomes a correctness problem.
(A note on that last one: an early version of the fan-out microbenchmark showed ReDevOps running serially, which turned out to be a configuration default, its safe-parallel scheduler shipped switched off. We fixed the default plumbing and refocused Family 2 on the safety semantics above; the microbenchmark did its real job, which was to catch the default.)
Guarantees have a price
Reproducibility, governance and safe execution are not free. The runtime records causal state, evaluates policy, verifies consequential actions and deliberately serializes operations when their resource scopes conflict. In those cases, lower raw throughput is an expected cost of stronger execution semantics, which the benchmark should show.
But that cost should not be generalized into "governed execution is slower." On conflict-free work, safe concurrency preserves parallelism, and on the bare orchestration benchmark ReDevOps is 2–2.3× lighter than LangGraph. The trade-off appears where the runtime intentionally does work a simpler executor does not: preserving evidence, enforcing authority, preventing conflicting effects, and making execution reproducible.
The relevant question is therefore "what guarantees were preserved at that execution speed?" Serialization caused by a real resource conflict is the scheduler choosing correctness over unsafe overlap, and Family 2 now makes that choice observable and explainable.
Beyond the benchmark: one runtime contract, three languages
The concurrency result is more useful if it is a runtime property. ReDevOps therefore implements the same resource/conflict-key contract independently in Python, Go, and Kotlin, with conformance suites exercising the same decisions: same key serializes, different keys may overlap, a bounded provider caps the wave, read-only work never locks, an unresolved key serializes conservatively, and the scheduler identifies the conflicting key. All three implementations pass the shared decision contract.
Conformance does not mean identical implementation, and on the load-bearing property, execution, all three now agree. Python and Kotlin execute the safe set concurrently and commit deterministically; Go, which previously made the same conflict-aware scheduling decision but executed the wave serially, now runs the wave in parallel too (opt-in, serial by default): operator invocations overlap behind a join barrier, then commit serially in deterministic node order, so committed state is identical to a serial run and only wall-clock changes. Bringing up its second concurrency ceiling, the Go runtime goes from 1.0× (serial) to 7.9× / 15.8× / 31.1× at 8 / 16 / 32 independent tools, on par with the concurrent executors, while keeping the conflict-aware serialization and the sub-microsecond decision path.
The important portability is execution semantics. A mission can carry the same resource-conflict rules, conservative fallback behavior, bounded concurrency and explainability across Python, Go and Kotlin implementations. For organizations with Python AI services, Go infrastructure and JVM business systems, governance does not have to change with the application language.
LangChain and LangGraph solve orchestration primarily in the Python application layer. ReDevOps is testing a different proposition: that the execution and governance contract can sit beneath heterogeneous applications and remain independently conformant across implementations.
The picture
Put the twelve families together and a clean line emerges:
Reach for a runtime when the hard part is everything around the call. ReDevOps wins where its governed, selective and replayable execution is the point: context selection (4, 5), selective recomputation (6), replay and recovery (7), capability routing (8), safety SLOs (9, 10, 11), and governed multi-step missions (12), while remaining lighter than LangGraph on raw orchestration (1). The Python, Go and Kotlin conformance suites add a separate result: these execution rules can be treated as a portable runtime contract.
Reach for LangChain/LangGraph, or nothing at all, when the workload is a straightforward chain or an embarrassingly parallel fan-out with no shared resources, no approvals, and no replay requirement. There the runtime's guarantees are overhead you're not using.
That is the honest frontier: a map of where a substrate beneath your agents changes the outcome, where a simpler flow already suffices, and (through the scheduler's EXPLAIN, evidence dependencies and mission ledger) why. Concurrency itself is commodity. Safe concurrency, recoverable execution, selective context and portable governance semantics are the runtime features, and where those guarantees require additional checks or deliberate serialization, the benchmark reports their cost alongside the benefit.
Reproducibility: all four adapters, the runner with full telemetry (RSS/USS, VRAM/KV-cache, time decomposition), and per-family workloads live in the benchmark repo. Each family carries its exact reproduce command. Model-free families (orchestration, concurrency, incremental, replay, mission) run anywhere; model-backed families need the shared vLLM endpoint. After the safe-concurrency change, including the Go parallel-execution work, the four families that touch the changed runtime code (1, 2, 7, 12) were re-run with no correctness discrepancy.