← redevops.io  /  Benchmarks  /  Fleet Backblaze Drive Stats · real fleet
Edge / fleet runtime · discovery · context · governance, at scale

The runtime, on a real device fleet

A fleet of devices is the forcing function for the runtime: discovery at scale, missions that span devices, reference-first context, and cross-device governance. Here those are measured on Backblaze Drive Stats — a real fleet of hundreds of thousands of drives with real failure ground truth — offline, before any hardware.

Evidence · controlled · internally reproduced 27 Oct 2026 on Backblaze Drive Stats: false alarms 229 to 97 (57.6% fewer, paired p effectively 0) with recall 0.807 to 0.756 (paired difference -5.1 points over 54 drives); the adaptive-routing frontier is unchanged (0.852 / 0.723 at 0.07 times the specialist calls); data-quality precision 0.892 to 0.300 to 0.850, and the 96-drive same-model batch is held for review.

The dataset & the test

Backblaze publishes daily SMART telemetry for its entire drive fleet, with a failure event on each drive's last day (Drive Stats, MIT). The scored slice is one quarter (Q2 2025) — 5,061 drives · 1,061 real failures (every failed drive's full history + a healthy sample). An independent oracle = the real failure events; nothing here consults the runtime to build the labels. Three tests: discover at-risk drives from the telemetry trajectory, pull context by reference, and let the real Governance Plane oversee the remediation of the whole fleet.

Discovery — prediction is an input, not the system

Couldn't a Random Forest do better than a hand-written rule? Probably — so we trained the alternatives a real fleet might already own (GradientBoosting, RandomForest, XGBoost) over the same leakage-safe SMART features, on held-out drives, and put them next to the rule. They cluster: the detector is a provider, not the differentiator.

detector · predict a real failure from SMART telemetryprecisionrecalllead time
Per-reading gateflag on any single nonzero sector — what a stateless per-action check sees0.7890.807—
Trajectory rule — inspectableflag on the rising trajectory, not one reading (best recall of the group)0.8920.75624 days
GradientBoostinglearned, held-out drives, same signals0.9100.698—
RandomForestdefault · tuned+balanced0.897 · 0.8390.682 · 0.739—
XGBoosttuned, held-out drives0.8500.714—
The detector is a providerOn the same signals the learned models cluster in a 0.84–0.91 precision band — none dominates the rule. Discovery treats any of them (RF, XGBoost, vendor model, LLM) as an interchangeable capability; the value is what happens after the signal.
58% fewer false alarmsReading the trajectory instead of one reading cuts false positives 229 → 97 at comparable recall, and flags failures 24 days early — historical context changes the decision, whatever detector you plug in.
The real ceiling is physical, ~0.81203 / 1,061 (19%) failures show no SMART warning at all — unpredictable by any detector, learned or hand-written; every model in the table hits this wall.

// 5,061 drives · 1,061 real failures · precision/recall vs the real failure oracle; lead time = median days from first flag to failure · reproduce: redevops-edge/fleet · python -m fleet.score

Optimal routing — the cheapest sufficient capability

If the detector is an interchangeable provider, Discovery's job is routing: run the cheap device-local rule first, accept it where confident, and escalate only the ambiguous middle band to a learned specialist. Measured on held-out drives — the decision-quality / resource frontier.

strategyprecisionrecallspecialist infer.telemetry read
Trajectory rule (device-local)no specialist calls0.9080.6540.00×0.60×
Always XGBoostspecialist on every drive0.8500.7141.00×1.00×
Always ensemble (3 models)every model, every drive0.8570.7173.00×1.00×
Discovery — adaptive routingspecialist only on the ambiguous 7%0.8520.7230.07×0.63×
Same decision, a fraction of the computeAdaptive routing matches — slightly beats — always-on XGBoost and the 3-model ensemble on precision and recall, running the specialist on 7% of drives: ~14× fewer specialist inferences than always-XGBoost, ~42× fewer than the ensemble.
Reference-first, one layer upContext Runtime refuses to move information the decision doesn't need; adaptive Discovery refuses to compute it. On a fleet or a power-budgeted Jetson, 7% vs 100% of the heavy inferences is the edge-vs-cloud line.
Modality-neutralThe unit routed over is a capability, not a model — heuristic, RandomForest, vendor diagnostic, OCR, VLM, human clarification, or a data-quality validator. Failure and data-quality discovery both have real numbers here (below); language and video are next.

// held-out 1,519 drives (318 fail) · specialist = XGBoost · inferences & telemetry as a fraction of an always-on learned pass · reproduce: redevops-edge/fleet · python -m fleet.score

Discovery before Discovery — can we trust the evidence?

There's a question logically before "is this drive failing?": is the telemetry even trustworthy? Data quality is another Discovery stage — the same optimizer over cheap null / range / reset validators, escalating only ambiguous records to a per-drive statistical specialist. Ground truth by disclosed defect injection, scored on a clean_universe() so Backblaze's natural artifacts aren't miscounted.

data-quality detector · 93,865 clean records · 4,709 injectedprecisionrecallspecialist calls
Cheap validators onlynull / range / reset — device-local1.0000.6590.00×
Always specialistper-drive statistical check on every record1.0000.9531.00×
Discovery — adaptiveescalate only the ambiguous 3%1.0000.9480.03×
The result that matters is downstreamDetecting bad data is table stakes. We corrupted the telemetry and re-ran the failure benchmark: precision collapsed 0.892 → 0.300. Running data-quality discovery first and withholding untrusted evidence recovered it to 0.850, recall unchanged (0.756 → 0.758).
Discovery is compositionalOne Discovery call establishes whether the evidence can be trusted before another uses it: DQ → failure → context → governance → mission. The optimizer didn't change; only the question did.
Same frontier, second domainAdaptive detection reaches the always-on specialist's quality (0.948 vs 0.953) at 3% of its calls — the routing result again, now over validators instead of failure detectors.

// disclosed defect injection (missing / out-of-range / reset / scale-drift) · scored on records verified clean in the raw data · reproduce: redevops-edge/fleet · python -m fleet.quality

Context & fleet governance — what only a runtime does

95.3%
fewer bytes moved. Reference-first context pulls only the flagged drives' last-30-day window and the fields the mission needs (1.1 MB) instead of shipping the whole fleet's telemetry (22.6 MB).
96 → held
cross-device governance. The biggest same-model batch flagged in a 7-day window was 96 drives. A per-drive check auto-quarantines all 96; the real Governance Plane correlates the same-model sequence and returns REQUIRE_REVIEW — a systemic sensor/firmware artifact isn't a fleet-wide failure.
What is and isn't claimed. The scored slice is ~21% failures by construction (every failed drive + a fixed healthy sample), so the absolute precision figures describe this slice, not fleet-wide prevalence — precision is prevalence-sensitive, and real fleets fail far more rarely. The like-for-like comparison (both methods score the same drives) is what the 58% fewer false alarms rests on. The trajectory rule also deliberately trades a little recall for precision (tunable) — fleets pay for false alarms, so precision is the right operating point. A true rate cap (">K in a window") is population mechanics — v0.2.x; the v0.2.0 plane here expresses the ordered same-model sequence, with the count reported as the population signal. And this validates the runtime's fleet capabilities over real telemetry — it is not a literal Jetson / on-device deployment. That (a minimal on-device runtime, then Jetson) is the next phase.

// reference-first context + the same-model mass-quarantine through the real Governance Plane · data: Backblaze Drive Stats (MIT) · code: redevops-edge/fleet

Out-of-sample — a different quarter

Every threshold and model above was chosen against Q2 2025, so reporting on Q2 is partly in-sample. We freeze all of it and evaluate on a held-out quarter — Q1 2025 (different drives, different time, no tuning contamination). If the numbers held only because they were fit here, this is where they'd fall apart.

metric · frozen on Q2, evaluated on Q1in-sample (Q2)out-of-sample (Q1)
Trajectory ruleprecision / recall / lead0.892 / 0.756 / 24d0.909 / 0.818 / 26d
Learned specialist (XGBoost)trained on Q2, tested on Q10.871 / 0.7470.848 / 0.771
Adaptive routingprecision / recall0.846 / 0.7470.847 / 0.777
Adaptive specialist callsfraction of drives escalated0.07×0.07×
Data-quality detection (adaptive)precision / recall1.000 / 0.9481.000 / 0.948
DQ downstream — failure precisionclean → corrupted → gated0.89 / 0.30 / 0.850.91 / 0.32 / 0.86
It holds — often betterThe Q2-chosen thresholds and Q2-trained models carry to a different quarter with no degradation (the trajectory rule is actually stronger on Q1). The results are out-of-sample-validated, not overfit to the tuning data.
The frozen escalation policy generalisesAdaptive routing hits the same 0.07× specialist-call fraction on unseen drives — the cheap-first/escalate decision boundary isn't quarter-specific.

// Q1 2025 · 5,067 drives · 1,067 real failures · frozen-on-Q2 evaluation · reproduce: FLEET_OOS=data/slice_oos.parquet python -m fleet.oos

// Backblaze Drive Stats is used only to benchmark the runtime's fleet capabilities — real fleet telemetry + real failure ground truth; cited per its attribution license, not redistributed. A balanced fixture is committed; the full run reproduces from the upstream quarterly archive.