Same Model, ~52% Fewer Blunders: Measuring What the Runtime Adds on Chess
We froze one base model and added the decision Runtime one layer at a time (verification, context, planning, bounded tools, discovery, persistent lessons) and scored every decision with a frozen Stockfish depth-16 oracle on 726 held-out chess positions the model never trained on. The full stack cuts the blunder rate 0.408 → 0.197 (−52%), same model throughout; and with the weights held fixed the held-out blunder rate falls 0.390 → 0.293 as the Runtime accumulates learned lessons.
The claim we actually wanted to test
It is easy to make an LLM look good at chess: hand it a strong engine and let the engine choose. That measures the engine, and it teaches you nothing about the runtime you would put under a real agent.
The claim we care about is narrower and harder:
The model supplies reasoning. The Runtime turns repeated decisions and their outcomes into a decision system that gets better, at a cost you can see.
To test that honestly you have to hold the model still. So we did. One base model, a local Qwen3.8-27B (a ~3B-active Qwen3 MoE at our GPU endpoint, temperature 0, fixed seed), is frozen for the entire experiment. Nothing about the weights changes, ever. The only thing that varies between arms is how much of the decision Runtime wraps the model: verification, retrieved context, an execution planner, a bounded tool, failure-pattern discovery, and a store of persistent learned lessons: the same components that run in production (context_runtime, the mission governance/evidence plane, PatternMemory, the lesson lifecycle).
Chess is the lab for the product. It gives us three things a real domain rarely offers at once: a fully known rule system, a powerful independent oracle to score decision quality, and unlimited repeatable trials. That lets us calibrate the runtime against ground truth before the stakes are real.
How a decision is scored
Every position is a single decision: given a legal board, produce a move. We score it with Stockfish 17.1 at depth 16, single-threaded and deterministic, used strictly as a measuring instrument for model decisions in the learning arms. For each move we record:
- Blunder rate: the fraction of moves that lose more than a threshold of centipawns versus the oracle's best move. The headline metric.
- Mean cp-loss: average centipawn loss against the oracle. The continuous version.
- Best-move rate: how often the move is the oracle's top choice.
- Cost: model calls and output tokens per decision.
The suite is 726 held-out positions and 1,203 training positions, drawn from a seeded corpus and stratified by game phase (opening / middlegame / endgame) and sharpness (quiet / tactical). Train and test are strictly isolated: no test outcome ever enters the learning store.
The ablation: one layer at a time
Each arm is the same frozen model plus a progressively larger slice of the Runtime. Lower blunder and cp-loss are better; higher best-move is better.
| Arm | Capability added | Blunder | cp-loss | Best-move | Calls | Tokens |
|---|---|---|---|---|---|---|
| A | LLM only + legal board | 0.408 | 304 | 0.163 | 1 | 6 |
| B | + deterministic legal-move verifier | 0.408 | 308 | 0.168 | 1 | 6 |
| C | + Context Runtime (history/examples) | 0.365 | 286 | 0.163 | 1 | 6 |
| D | + Execution Planner (candidate + critique) | 0.402 | 296 | 0.175 | 2 | 49 |
| E | + bounded engine hint (shallow, top-k) | 0.167 | 126 | 0.375 | 2 | 47 |
| F | + Discovery (failure-pattern reflection) | 0.380 | 288 | 0.167 | 2 | 56 |
| G | + persistent learned lessons | 0.307 | 242 | 0.152 | 2 | 65 |
| H | Complete Runtime stack | 0.197 | 174 | 0.270 | 2 | 50 |
| I | H + bounded engine at decision time | 0.200 | 170 | 0.264 | 2 | 50 |
Four things are worth saying plainly, including the null result:
Verification is a safety property. A → B is essentially flat. This model already plays legal moves, so a legal-move verifier changes the illegal-move rate (which matters enormously for an agent taking real actions), leaving the quality of the moves it was already going to make flat. We report the null result because it is the honest one: verification earns its place by preventing invalid actions, and this benchmark can't see that value because the model rarely produces one.
Context and learning move the needle with no tool at all. C (retrieved history and examples) and G (mined lessons) each cut blunders using the same one or two model calls, no engine, no extra weights. That is the part that transfers to domains where no oracle exists.
A bounded tool is the single biggest lever. Arm E adds a shallow, top-k engine hint and roughly halves the blunder rate. We label it as exactly that: external assistance, clearly separated from the model's own judgment. In a real domain this is the "call the pricing service / the risk model" step, powerful, and honestly attributed.
The stack compounds. H combines context, planning, lessons, and a bounded hint into the best all-round arm at two model calls: a 52% blunder reduction over the naked model. Adding more engine at decision time (I) barely moves it, which tells us the stack is doing the work.
The result the whole design exists to isolate
The ablation shows what each layer adds. This next table shows the model learning, with its weights frozen.
We hold the model fixed and let only the Runtime's accumulated experience change. Discovery mines lessons from scored training decisions; the Learn layer retrieves them at decision time. Then we re-score on the same held-out suite as training accrues:
| Training decisions | Usable lessons | Blunder (held-out) | cp-loss |
|---|---|---|---|
| 0 | 0 | 0.390 | 289 |
| 50 | 5 | 0.383 | 286 |
| 150 | 11 | 0.351 | 269 |
| 400 | 13 | 0.293 | 254 |
A monotonic drop in held-out blunder rate as experience accumulates: no fine-tuning, no weight change. The same model, given a Runtime that remembers what went wrong, plays measurably better. The improvement belongs to the Runtime alone.
We are deliberately unglamorous about what the lessons are: per-signature move-category priors, "in this kind of position, be cautious with queen checks", mined from scored failures and retrieved by position signature. Not a learned evaluation function, not a fine-tune. Coarse, inspectable priors. That they produce a monotonic, held-out improvement is exactly the modest, real result we wanted.
Where it's hard
The full-stack arm on the held-out suite, split by position type:
| Position type | n | Blunder | cp-loss |
|---|---|---|---|
| opening · quiet | 110 | 0.045 | 57 |
| middlegame · quiet | 382 | 0.162 | 154 |
| endgame · quiet | 47 | 0.191 | 156 |
| opening · tactical | 20 | 0.250 | 156 |
| middlegame · tactical | 156 | 0.372 | 285 |
| endgame · tactical | 11 | 0.364 | 550 |
Quiet openings are nearly solved; sharp tactical positions remain the hardest: exactly where a shallow model most needs the Runtime, and exactly where the honest headroom is. We show you the 0.372 plainly.
What this is
- One model, one run. A single frozen base model and a single publish run on 726 held-out / 1,203 training positions. It establishes direction and cost, with no leaderboard claim.
- The oracle only measures. Stockfish scores decision quality; it is never the principal decision-maker in the learning arms. The E/I arms use a separate, deliberately shallow hint, always labelled as external.
- Every decision is retained. All arms' decisions, the mined lessons, the learning curve, and a frozen manifest (model, oracle, seeds, git commit, host) are kept so the run is replayable, and failures stay in the dataset.
The lesson we're carrying to real work: a correction is a candidate
Chess is generous in one specific way, and it is worth being honest about it: Stockfish is a clean, independent outcome signal. Every decision gets scored against ground truth, cheaply and immediately. That is why chess is the laboratory, and it is exactly what a real business workflow does not have.
So the tempting shortcut (a human corrects the system once, and that correction becomes a permanent rule) is precisely the thing this benchmark argues against. Note what actually produced the learning-curve result: the lessons were not written from single games. They were mined from many scored decisions, gated by support and confidence, and only the ones that held up were retained (20 usable lessons out of 29 candidate signatures, mean confidence 0.80). A one-off correction, in that regime, is still a candidate.
That generalizes into the lifecycle we're carrying into the real domains:
human correction
↓ observation (what happened, in what context)
↓ learning candidate (a proposed change)
↓ supporting examples / outcomes
↓ confidence
↓ proposed amendment
↓ review / shadow
↓ workflow vN+1
We carry that full lifecycle. When the oracle is weak (which is the normal case outside chess), the review/shadow step is where a human, or a slower trusted signal, stands in for Stockfish.
It also points at a single Learn contract we can share across every domain and every app:
Experience context_signature · decision · action · outcome · evidence_refs · runtime_version
Pattern scope · supporting_experiences[] · counterexamples[] · confidence
Lesson rule / prior / strategy · applicability · provenance · lifecycle
Candidate proposed_change · supporting_pattern · validation_required
Same lifecycle, different products. Chess produces "in this position signature, be cautious with queen checks." Revenue will eventually produce "opportunities with characteristics X and Y have historically responded poorly." Projects can produce "when exactly one contact is associated with the active opportunity, select that contact." Different domains; one Runtime learning contract.
And it composes cleanly with demonstration-derived workflows: demonstration teaches the Runtime the initial workflow; experience teaches it where that workflow should improve. That is a considerably stronger position than "record a workflow and replay it", and it is the honest version of self-improvement, because every change is a candidate that has to earn its place.
Why chess, and what comes next
Chess is the first lab because it is unforgiving and cheap: a known rule system, an independent oracle, and unlimited trials. Those let us calibrate the Runtime against ground truth in a place where being wrong costs nothing.
The same contracts (decision, evidence, discovery, persistent lessons) are built to move to economically consequential environments next: simulated auctions (cost, competition, deadlines, delayed outcomes), then real revenue decisions: which lead, which tender, which outreach deserves the next unit of scarce attention. There the oracle gets weaker and the stakes get real, which is the whole point of proving the mechanism here first.
The claim we set out to test holds, at the modest scale we tested it: hold the model fixed, and the Runtime measurably improves the decisions, and keeps improving them as it accumulates experience, at a cost you can read off the table.
The full ablation, learning curve, per-stratum breakdown, and reproduction details are on the benchmark page.