A fleet of devices is the forcing function for the runtime: discovery at scale, missions that span devices, reference-first context, and cross-device governance. Here those are measured on Backblaze Drive Stats — a real fleet of hundreds of thousands of drives with real failure ground truth — offline, before any hardware.
Evidence · controlled · internally reproduced 27 Oct 2026 on Backblaze Drive Stats: false alarms 229 to 97 (57.6% fewer, paired p effectively 0) with recall 0.807 to 0.756 (paired difference -5.1 points over 54 drives); the adaptive-routing frontier is unchanged (0.852 / 0.723 at 0.07 times the specialist calls); data-quality precision 0.892 to 0.300 to 0.850, and the 96-drive same-model batch is held for review.
Backblaze publishes daily SMART telemetry for its entire drive fleet, with a failure event on each drive's last day (Drive Stats, MIT). The scored slice is one quarter (Q2 2025) — 5,061 drives · 1,061 real failures (every failed drive's full history + a healthy sample). An independent oracle = the real failure events; nothing here consults the runtime to build the labels. Three tests: discover at-risk drives from the telemetry trajectory, pull context by reference, and let the real Governance Plane oversee the remediation of the whole fleet.
Couldn't a Random Forest do better than a hand-written rule? Probably — so we trained the alternatives a real fleet might already own (GradientBoosting, RandomForest, XGBoost) over the same leakage-safe SMART features, on held-out drives, and put them next to the rule. They cluster: the detector is a provider, not the differentiator.
| detector · predict a real failure from SMART telemetry | precision | recall | lead time |
|---|---|---|---|
| Per-reading gateflag on any single nonzero sector — what a stateless per-action check sees | 0.789 | 0.807 | — |
| Trajectory rule — inspectableflag on the rising trajectory, not one reading (best recall of the group) | 0.892 | 0.756 | 24 days |
| GradientBoostinglearned, held-out drives, same signals | 0.910 | 0.698 | — |
| RandomForestdefault · tuned+balanced | 0.897 · 0.839 | 0.682 · 0.739 | — |
| XGBoosttuned, held-out drives | 0.850 | 0.714 | — |
// 5,061 drives · 1,061 real failures · precision/recall vs the real failure oracle; lead time = median days from first flag to failure · reproduce: redevops-edge/fleet · python -m fleet.score
If the detector is an interchangeable provider, Discovery's job is routing: run the cheap device-local rule first, accept it where confident, and escalate only the ambiguous middle band to a learned specialist. Measured on held-out drives — the decision-quality / resource frontier.
| strategy | precision | recall | specialist infer. | telemetry read |
|---|---|---|---|---|
| Trajectory rule (device-local)no specialist calls | 0.908 | 0.654 | 0.00× | 0.60× |
| Always XGBoostspecialist on every drive | 0.850 | 0.714 | 1.00× | 1.00× |
| Always ensemble (3 models)every model, every drive | 0.857 | 0.717 | 3.00× | 1.00× |
| Discovery — adaptive routingspecialist only on the ambiguous 7% | 0.852 | 0.723 | 0.07× | 0.63× |
// held-out 1,519 drives (318 fail) · specialist = XGBoost · inferences & telemetry as a fraction of an always-on learned pass · reproduce: redevops-edge/fleet · python -m fleet.score
There's a question logically before "is this drive failing?": is the telemetry even trustworthy? Data quality is another Discovery stage — the same optimizer over cheap null / range / reset validators, escalating only ambiguous records to a per-drive statistical specialist. Ground truth by disclosed defect injection, scored on a clean_universe() so Backblaze's natural artifacts aren't miscounted.
| data-quality detector · 93,865 clean records · 4,709 injected | precision | recall | specialist calls |
|---|---|---|---|
| Cheap validators onlynull / range / reset — device-local | 1.000 | 0.659 | 0.00× |
| Always specialistper-drive statistical check on every record | 1.000 | 0.953 | 1.00× |
| Discovery — adaptiveescalate only the ambiguous 3% | 1.000 | 0.948 | 0.03× |
// disclosed defect injection (missing / out-of-range / reset / scale-drift) · scored on records verified clean in the raw data · reproduce: redevops-edge/fleet · python -m fleet.quality
// reference-first context + the same-model mass-quarantine through the real Governance Plane · data: Backblaze Drive Stats (MIT) · code: redevops-edge/fleet
Every threshold and model above was chosen against Q2 2025, so reporting on Q2 is partly in-sample. We freeze all of it and evaluate on a held-out quarter — Q1 2025 (different drives, different time, no tuning contamination). If the numbers held only because they were fit here, this is where they'd fall apart.
| metric · frozen on Q2, evaluated on Q1 | in-sample (Q2) | out-of-sample (Q1) |
|---|---|---|
| Trajectory ruleprecision / recall / lead | 0.892 / 0.756 / 24d | 0.909 / 0.818 / 26d |
| Learned specialist (XGBoost)trained on Q2, tested on Q1 | 0.871 / 0.747 | 0.848 / 0.771 |
| Adaptive routingprecision / recall | 0.846 / 0.747 | 0.847 / 0.777 |
| Adaptive specialist callsfraction of drives escalated | 0.07× | 0.07× |
| Data-quality detection (adaptive)precision / recall | 1.000 / 0.948 | 1.000 / 0.948 |
| DQ downstream — failure precisionclean → corrupted → gated | 0.89 / 0.30 / 0.85 | 0.91 / 0.32 / 0.86 |
// Q1 2025 · 5,067 drives · 1,067 real failures · frozen-on-Q2 evaluation · reproduce: FLEET_OOS=data/slice_oos.parquet python -m fleet.oos
// Backblaze Drive Stats is used only to benchmark the runtime's fleet capabilities — real fleet telemetry + real failure ground truth; cited per its attribution license, not redistributed. A balanced fixture is committed; the full run reproduces from the upstream quarterly archive.