← redevops.io  /  Benchmarks  /  Self-Learning Decision Runtime · Chess Qwen3.8-27B (frozen) · Stockfish 17.1 depth-16
Decision runtime · measured, held-out

Same model, fewer blunders

The claim under test is not "an LLM plus a chess engine can win games." It is: the model supplies reasoning; the Runtime turns repeated decisions and their outcomes into an improving decision system — at a measured cost. So we froze the base model and, arm by arm, added one Runtime layer at a time — verification, context, planning, bounded tools, discovery, persistent learning — and scored every decision with a strong, frozen Stockfish oracle on 726 positions the model never trained on.

Baseline (naked model)
0.408blunder rate — arm A, one direct move
Full Runtime stack
0.197blunder rate — arm H (−52% vs A), same frozen model
Learning, model fixed
0.390→0.293held-out blunder rate as the Runtime accrues 13 lessons
Cost of the stack
2 callsvs 1 — ~50 output tokens vs 6 (quality is not free)

Same weights, same opponents, same budget. The only thing that changed between arms was how much of the Runtime wrapped the model — and the blunder rate fell by half.

The ablation: what each layer adds, and what it costs

Every arm is the same Qwen3.8-27B plus a progressively larger slice of the real Runtime — the same components used in production (context_runtime, the mission governance/evidence plane, PatternMemory, the lesson lifecycle), not chess-only reimplementations. Scored on the frozen held-out suite; lower blunder rate and cp-loss are better, higher best-move is better, calls/tokens are the cost.

ArmCapability addedBlunder rateMean cp-lossBest-moveModel callsOut tokens
ALLM only + legal board0.4083040.16316
B+ deterministic legal-move verifier0.4083080.16816
C+ Context Runtime (history/examples)0.3652860.16316
D+ Execution Planner (candidate + critique)0.4022960.175249
E+ bounded engine hint (shallow, top-k)0.1671260.375247
F+ Discovery (failure-pattern reflection)0.3802880.167256
G+ persistent learned lessons0.3072420.152265
HComplete Runtime stack0.1971740.270250
IH + bounded engine at decision time0.2001700.264250
Verification is safety, not strength. A→B is ~flat: this model already plays legal moves, so the verifier changes the illegal-move rate, not the quality. An honest null result.
Context and learning move the needle without tools. C (retrieved history) and G (mined lessons) each cut blunders with the same one/two model calls — no engine, no extra weights.
A bounded tool is the biggest single lever. E adds a shallow, top-k engine hint and roughly halves blunders — clearly labelled as external assistance, not the model.
The stack compounds. H combines context + planning + lessons + a bounded hint into the best all-round result at two model calls; adding more engine at decision time (I) barely moves it.

The model is frozen. The system still learns.

This is the result the whole design exists to isolate. We hold the weights fixed and let only the Runtime's accumulated experience change: Discovery mines lessons from scored training decisions, and the Learn layer retrieves them at decision time. Re-scored on the same held-out suite as training accrues — with strict train/test isolation, so no test outcome ever enters the learning store.

Training decisionsUsable lessonsBlunder rate (held-out)Mean cp-loss
000.390289
5050.383286
150110.351269
400130.293254

A monotonic drop in blunder rate with accumulated experience — no fine-tuning, no weight change. The improvement is the Runtime's, not the model's.

Where it's hard: by position type

The full-stack arm (H) on the held-out suite, split by phase and sharpness. Quiet openings are nearly solved; sharp tactical positions remain the hardest — exactly where a shallow model most needs the Runtime, and where honest headroom remains.

Position typenBlunder rateMean cp-loss
opening · quiet1100.04557
middlegame · quiet3820.162154
endgame · quiet470.191156
opening · tactical200.250156
middlegame · tactical1560.372285
endgame · tactical110.364550

What this is — and isn't

One model, one run. A single frozen base model (Qwen3.8-27B) and a single publish run on 726 held-out / 1,203 training positions. It establishes direction and cost, not a leaderboard.
The oracle only measures; a weak opponent lets progress show. Stockfish (depth-16, deterministic) scores decision quality; it is never the principal decision-maker in the learning arms. Arms E/I use a separate, deliberately shallow hint, always labelled.
Lessons are coarse priors. The learned lessons are per-signature move-category priors (e.g. "in this position type, be cautious with queen checks"), not a learned evaluation function. Modest, monotonic, honest.
Every decision is evidence. Every arm's decisions, the mined lessons, the frozen manifest (model, oracle, seeds, git commit) and the learning curve are retained — the run is replayable, and failures stay in the dataset.
Frozen base model Qwen3.8-27B — a ~3B-active Qwen3 MoE at the local GPU endpoint (temp 0, seed 12648430) · Stockfish 17.1 depth-16, Threads=1 (deterministic) · 726 held-out / 1,203 training positions, seeded corpus · train/test isolation enforced · full ablation, learning curve, mined lessons and provenance manifest (model, oracle, seeds, git 2279e76) retained per run.