The claim under test is not "an LLM plus a chess engine can win games." It is: the model supplies reasoning; the Runtime turns repeated decisions and their outcomes into an improving decision system — at a measured cost. So we froze the base model and, arm by arm, added one Runtime layer at a time — verification, context, planning, bounded tools, discovery, persistent learning — and scored every decision with a strong, frozen Stockfish oracle on 726 positions the model never trained on.
Same weights, same opponents, same budget. The only thing that changed between arms was how much of the Runtime wrapped the model — and the blunder rate fell by half.
Every arm is the same Qwen3.8-27B plus a progressively larger slice of the real Runtime — the same components used in production (context_runtime, the mission governance/evidence plane, PatternMemory, the lesson lifecycle), not chess-only reimplementations. Scored on the frozen held-out suite; lower blunder rate and cp-loss are better, higher best-move is better, calls/tokens are the cost.
| Arm | Capability added | Blunder rate | Mean cp-loss | Best-move | Model calls | Out tokens |
|---|---|---|---|---|---|---|
| A | LLM only + legal board | 0.408 | 304 | 0.163 | 1 | 6 |
| B | + deterministic legal-move verifier | 0.408 | 308 | 0.168 | 1 | 6 |
| C | + Context Runtime (history/examples) | 0.365 | 286 | 0.163 | 1 | 6 |
| D | + Execution Planner (candidate + critique) | 0.402 | 296 | 0.175 | 2 | 49 |
| E | + bounded engine hint (shallow, top-k) | 0.167 | 126 | 0.375 | 2 | 47 |
| F | + Discovery (failure-pattern reflection) | 0.380 | 288 | 0.167 | 2 | 56 |
| G | + persistent learned lessons | 0.307 | 242 | 0.152 | 2 | 65 |
| H | Complete Runtime stack | 0.197 | 174 | 0.270 | 2 | 50 |
| I | H + bounded engine at decision time | 0.200 | 170 | 0.264 | 2 | 50 |
This is the result the whole design exists to isolate. We hold the weights fixed and let only the Runtime's accumulated experience change: Discovery mines lessons from scored training decisions, and the Learn layer retrieves them at decision time. Re-scored on the same held-out suite as training accrues — with strict train/test isolation, so no test outcome ever enters the learning store.
| Training decisions | Usable lessons | Blunder rate (held-out) | Mean cp-loss |
|---|---|---|---|
| 0 | 0 | 0.390 | 289 |
| 50 | 5 | 0.383 | 286 |
| 150 | 11 | 0.351 | 269 |
| 400 | 13 | 0.293 | 254 |
A monotonic drop in blunder rate with accumulated experience — no fine-tuning, no weight change. The improvement is the Runtime's, not the model's.
The full-stack arm (H) on the held-out suite, split by phase and sharpness. Quiet openings are nearly solved; sharp tactical positions remain the hardest — exactly where a shallow model most needs the Runtime, and where honest headroom remains.
| Position type | n | Blunder rate | Mean cp-loss |
|---|---|---|---|
| opening · quiet | 110 | 0.045 | 57 |
| middlegame · quiet | 382 | 0.162 | 154 |
| endgame · quiet | 47 | 0.191 | 156 |
| opening · tactical | 20 | 0.250 | 156 |
| middlegame · tactical | 156 | 0.372 | 285 |
| endgame · tactical | 11 | 0.364 | 550 |
The same contracts — decision, evidence, discovery, persistent lessons — are built to move to economically consequential environments next: simulated auctions (cost, competition, deadlines, delayed outcomes), then real revenue decisions (which lead, which tender, which outreach deserves the next unit of scarce attention). Chess gives what those rarely do: a known rule system, a powerful independent oracle, and unlimited repeatable trials to calibrate against before the stakes are real.
The lesson we carry out of the lab: a human correction is not a rule. Note the learning curve above came from lessons mined from many scored decisions and gated by support and confidence — not written from single games. Outside chess the oracle is weak, so a correction enters a lifecycle — observation → candidate → supporting outcomes → confidence → review / shadow → next workflow version — rather than becoming law on the spot. Demonstration teaches the Runtime the initial workflow; experience teaches it where that workflow should improve.