All articles
    ai-runtime
    edge
    fleet
    discovery-runtime
    context-runtime
    governance
    backblaze
    smart
    predictive-maintenance
    benchmark
    llmops

    The Runtime, on a Real Fleet: Testing Fleet-Scale AI on 1,061 Real Drive Failures

    August 19, 2026redevops.io
    The Runtime, on a Real Fleet: Testing Fleet-Scale AI on 1,061 Real Drive Failures

    Part of the edge/distributed-runtime thread. The edge is a forcing function. A fleet of devices demands exactly the capabilities an AI runtime is supposed to have and a per-action agent stack cannot fake: discovery at scale, context you address in place, and governance that reasons across devices as a whole. This is those capabilities, measured on a real fleet, before any hardware.

    Most "edge AI" demos start by buying a device. We started somewhere cheaper and more honest: a real fleet with real failure ground truth, and a question: does the runtime actually do the three things a fleet needs that a single-device, single-action agent cannot?

    The three things:

    1. Discovery at scale: surface the drives that are actually getting worse, out of hundreds of thousands, without drowning an operator in false alarms.
    2. Reference-first context: when a mission needs a drive's history, address the slice it needs from the fleet.
    3. Cross-device governance: when a thousand drives of the same model light up in the same week, recognize that as a systemic signal and hold the mass action for review, instead of auto-remediating a thousand times in a row.

    None of these is a language-model trick. They are runtime properties. And they happen to be measurable, offline, on data that already exists.

    The dataset: a real fleet that already tells the truth

    A real fleet of drives, each emitting a telemetry trajectory over time
    Scroll the diagram sideways to read itFull size

    Backblaze Drive Stats is one of the most useful public datasets in infrastructure: every day, Backblaze records the SMART telemetry of its entire hard-drive fleet (hundreds of thousands of drives) and stamps a failure event on each drive's last day of service. That failure column is the thing that makes this a benchmark: it is independent ground truth. Nothing in our runtime helps produce the labels; the drives failed in Backblaze's data centers, and we score against what actually happened.

    We took one quarter (Q2 2025) and built a scored slice: 5,061 drives, 1,061 of which really failed (every failed drive's full history, plus a fixed healthy comparison sample). One caveat up front, because precision is prevalence-sensitive: that ~21% failure rate is a property of this benchmark slice; Backblaze's fleet has far lower real-world prevalence. So the absolute precision numbers below describe the slice. On a fleet where failures are rarer, absolute precision would be lower for every method. What does carry over is the comparison: the per-reading gate and the trajectory view score the exact same drives, so the 58%-fewer-false-alarms result is like-for-like. SMART gives us the physical signals that matter: reallocated sectors, pending and offline-uncorrectable sectors, reported-uncorrectable errors, command timeouts, power-on hours, temperature. The oracle is the real failure date. Everything below is measured against it.

    Backblaze publishes this under a license that asks you to cite them and build on it, not resell the raw data, which is exactly what we do. We use it only to benchmark the runtime; we don't ship it.

    Discovery: prediction is an input, not the system

    Before and after: on the same 5,061 drives, a per-reading gate raises 229 false alarms; the trajectory view raises 97 (58% fewer) at comparable recall, with 24 days of warning before failure.
    Scroll the diagram sideways to read itFull size

    Let me start with the objection an ML engineer raises first, because dodging it would make everything after it suspect. Couldn't a Random Forest predict drive failures better than a hand-written rule? Probably, and that is exactly the point.

    ReDevOps is not a hard-drive-failure model. A real fleet very likely already has one: a Random Forest, a tuned XGBoost model, a vendor health score, an anomaly detector, a device-specific diagnostic. So we trained the alternatives ourselves (GradientBoosting, RandomForest, XGBoost) over the same leakage-safe SMART features, scored on held-out drives, and put their numbers directly next to the rule:

    detector (same SMART signals)precisionrecalllead time
    per-reading gate (stateless baseline)0.7890.807n/a
    trajectory rule (inspectable)0.8920.75624 days
    GradientBoosting0.9100.698n/a
    RandomForest (default)0.8970.682n/a
    RandomForest (tuned)0.8390.739n/a
    XGBoost (tuned)0.8500.714n/a

    They cluster. On the same signals nothing runs away with it: the learned models sit in a 0.84–0.91 precision band, trading a little precision for a little recall as you tune them, and the cheap, inspectable trajectory rule actually posts the best recall of the group. So the choice of detector is close to a wash, and, more importantly, it is not where the runtime earns its keep. Give a production fleet richer features and a stronger model and it may well pull ahead of our rule; good. The argument here does not depend on the rule winning, because the rule is not the product.

    Discovery Runtime treats every one of these as an interchangeable capability: heuristic, Random Forest, XGBoost, vendor model, an LLM/VLM, an ensemble. Its job is to use them. It is to decide what evidence is relevant, maintain the trajectory that makes any of them meaningful, invoke whichever detector is best for the decision, and turn its raw output (P(failure) = 0.87) into a verified, canonical, actionable signal with enough provenance attached for everything downstream to act on. The trajectory rule in this benchmark is deliberately simple and readable precisely so it can't be mistaken for the differentiator.

    That said, how the trajectory beats the naive gate is worth seeing, because it's the same lesson in miniature. The stateless baseline (what a per-action check sees) flags any drive the moment a single day shows a nonzero sector fault. It scores 0.789/0.807; the recall is fine, the precision is the problem, because a drive can carry a few reallocated sectors stably for months and never fail. The rule instead reads the series: pending/offline-uncorrectable sectors appearing (rare and strongly predictive), or reallocated sectors rising above the drive's own baseline (climbing above a stable count). That single change cuts false alarms from 229 to 97 (58% fewer) at comparable recall, 24 days before failure. The same result as our out-of-order rendering and governance work, in a third domain: historical context materially changes the decision. Whatever detector you drop in, it needs that trajectory to stand on, and maintaining it is Discovery's job.

    Discovery's real job: route to the cheapest sufficient capability

    If the detector is an interchangeable provider, the interesting question stops being which model and becomes which capability, for which drive, and when is a cheap answer already enough? That is a routing problem, the same one Context Runtime solves for retrieval and the Planner solves for execution: acquire the minimum sufficient evidence for the decision, and no more.

    So we built it and measured it. The cheap, device-local trajectory capability runs first, on every drive. Where it is confident (a clear pending/offline signal, sectors clearly climbing, or a drive with no fault signal at all) its verdict stands and nothing expensive runs. Only on the ambiguous middle band (a little reallocation that isn't clearly rising, a stray timeout) does Discovery escalate that drive to a learned specialist (here, XGBoost). The two responsibilities stay separate, deliberately: the optimizer decides what evidence to acquire next; a fusion step decides what the evidence means.

    On the held-out drives, the frontier comes out like this:

    strategyprecisionrecallspecialist inferencestelemetry read
    trajectory rule (device-local)0.9080.6540.00×0.60×
    always XGBoost0.8500.7141.00×1.00×
    always ensemble (3 models)0.8570.7173.00×1.00×
    Discovery: adaptive routing0.8520.7230.07×0.63×

    Read the bottom row against the two above it. Adaptive routing matches (slightly beats) the always-on XGBoost and the three-model ensemble on both precision and recall, while running the expensive specialist on just 7% of drives and reading about two-thirds of the telemetry. Roughly fourteen times fewer specialist inferences than always-XGBoost, more than forty times fewer than the ensemble: same decision. (The specialist-inference column counts learned-model calls only; the cheap device-local trajectory pass runs on every drive but isn't one.)

    That is what a discovery runtime is for, stated as a number. We spend the expensive capability only where the decision is genuinely in doubt, and the comparison showed a better failure model is unnecessary. It's reference-first context's principle moved one layer up: Context Runtime refuses to move information the decision doesn't need; adaptive Discovery refuses to compute it. On a hundred-thousand-device fleet, or a Jetson with a power budget, 7% versus 100% of the heavy inferences is the difference between a runtime that fits on the edge and one that doesn't.

    And none of this is specific to drives. The unit the optimizer routes over is a capability, modality-neutral by design: a heuristic, a Random Forest, a vendor diagnostic, an OCR pass, a temporal video model, a VLM, a human clarification. The same cheap-first, escalate-on-ambiguity loop is how Discovery would turn ambiguous natural-language intent into a verified canonical form, or find the few seconds in fourteen hours of video worth running an expensive temporal model over. Those remain the direction, awaiting measurement; one more domain we can already put real numbers on sits logically before everything above: whether the evidence itself can be trusted at all.

    Discovery is compositional: can we trust the evidence?

    Failure discovery asks whether a drive is becoming unsafe. But there is a question logically before it: is the telemetry even trustworthy enough to make that call? We built data quality as another Discovery stage of the same runtime. The same capability optimizer runs cheap null, range, and reset validators first; only the ambiguous records escalate to a per-drive statistical specialist.

    On 93,865 genuinely clean held-out Backblaze records seeded with 4,709 disclosed defects (missing values, out-of-range values, resets, scale drift) the cheap validators alone reach 1.000 precision / 0.659 recall; running the specialist on every record reaches 1.000 / 0.953; and adaptive Discovery reaches 1.000 / 0.948 while invoking the specialist on just 3% of records. The routing frontier again, in a second domain.

    data-quality detectorprecisionrecallspecialist calls
    cheap validators only1.0000.6590.00×
    always specialist (on all)1.0000.9531.00×
    Discovery adaptive1.0000.9480.03×

    But detecting bad data isn't the result that matters most. We corrupted the telemetry and re-ran the failure benchmark. Failure precision collapsed from 0.892 to 0.300: scale-drift spikes fire the trajectory detector on healthy drives. Running data-quality discovery first and withholding the evidence it couldn't trust recovered precision to 0.850, with recall effectively unchanged: 0.756 → 0.758. Discovering whether the evidence was trustworthy measurably rescued the discovery that depended on it.

    One qualification on the ground truth, because it matters. Backblaze gives us real telemetry and real downstream failure events, but it does not label corrupted records. So data-quality ground truth comes from disclosed defect injection at known positions, scored only against a clean_universe() of records verified clean in the raw data, so Backblaze's own naturally-occurring artifacts are never counted as injected defects.

    That gives Discovery a compositional form: one Discovery call establishes whether the evidence can be trusted before another Discovery call uses that evidence to establish something about the world (data-quality discovery → failure discovery → context → governance → mission). The optimizer didn't change; only the question did.

    (A rigor note, in the spirit of the leakage catch below. Building this benchmark caught two of our own mistakes before it caught anything about drives: a monotonicity check that cascaded (one scale-drift spike inflating a running maximum until every later normal record looked like a drop) and an evaluation universe that would have scored Backblaze's natural artifacts as if they were injected defects. Both are fixed above. It's becoming a recurring theme: the benchmark is here to find where the runtime (or the experiment) is wrong.)

    The one real ceiling is physical

    Notice what the comparator table does not contain: a model that clears ~0.81 recall. That's a wall no tuning clears, and it's worth naming precisely because it's the honest limit of the whole exercise.

    Of the 1,061 real failures, 203 (19%) show no SMART warning at all before they fail: no reallocated sectors, no pending sectors, nothing. Electronics failures, controller failures, sudden mechanical death: unpredictable from SMART by any detector, learned or hand-written. Every model in the table runs into it (that's why none exceeds ~0.74 recall here), which bounds recall at roughly 0.81 no matter what you plug in. We say that plainly: a fleet system cannot see failures that leave no trace.

    (A note on rigor, because it nearly bit us: our first learned model scored 0.997 recall, which should make anyone suspicious. It was leakage: features derived from power-on-hours and days-observed encode the label, because a failed drive's history is truncated at failure while a healthy drive runs the whole quarter. "Few days observed → failed" is the answer written on the back of the card. We removed those features and kept only the physical SMART signals; the number dropped to the honest figures above. If a benchmark number looks too good, it usually is, and it's why every model in the comparator table is scored the same leakage-safe way.)

    Context: address the slice the mission needs

    When a mission wants to act on the flagged drives (open a ticket, stage a migration, brief an operator) it needs their telemetry. The lazy way is to hand the whole fleet's daily SMART table to whatever's reasoning about it. The runtime does the opposite: it pulls only the flagged drives' last-30-day window, and only the fields the mission actually needs.

    On the slice that's 22.6 MB shipped-everything versus 1.1 MB by-reference (95.3% fewer bytes moved). This is the Context Runtime thesis applied to telemetry: computed context is an addressable artifact; you reference the slice you need, with the whole store staying in place. At a single drive it's a rounding error. At fleet scale, across every mission, every day, it's the difference between a system that runs at the edge and one that only runs where bandwidth is free.

    Governance: 96 drives of the same model, all 96 as one signal

    Now the failure mode that only exists at fleet scale. In one 7-day window, the biggest batch of same-model drives our discovery flagged was 96 drives, all one HGST model.

    A per-drive check does the obvious, wrong thing: it quarantines all 96, one decision at a time, each one locally reasonable. But 96 same-model drives degrading in the same week is one signal across all 96: a firmware regression, a bad batch, a sensor artifact, a data-center environmental event. Auto-remediating all 96 could pull a huge slice of capacity offline over what turns out to be a telemetry glitch.

    We ran this through the real ReDevOps Governance Plane: the same plane behind our trajectory-governance benchmark, imported from the actual runtime, not a mock. It correlates the same-model quarantine sequence across devices and returns REQUIRE_REVIEW, enforceable, citing the events that triggered it. Here's the measured difference, stated plainly: per-drive evaluation did not capture the correlated 96-drive condition (each action looked permissible on its own) while the Governance Plane, reasoning over the trajectory, did. Ninety-six individually permissible actions became one fleet-level REQUIRE_REVIEW: this looks systemic: a human should look before we act at scale.

    One disclosure here, because we hold ourselves to it elsewhere: a true rate cap (">K of the same model in a window" as a hard population limit) is population mechanics, which is still on our roadmap (population controls). The Governance Plane expresses the ordered same-model sequence and reports the population count alongside as the signal. It's the right decision for the right reason today; the hard cap is a near-term addition, not a claim we're making now.

    Prediction is one capability; the runtime is the system

    Put the whole thing on one table and the argument resolves. Score the operational system as a whole, and a better predictor doesn't move the columns that matter:

    systemdetect failurecontext-efficientcorrelate the fleetgovern the actionexecute safely
    per-reading SMART rule✓n/a✗✗✗
    learned model (RF / XGBoost / GBM)✓✓n/a✗✗✗
    ReDevOps + any predictor✓✓✓✓✓✓

    A Random Forest that beats our rule makes this argument stronger, because the runtime is model-agnostic: it slots the better detector into the detect column and everything to its right still only exists in the bottom row. The predictor hands you Drive X: P(failure) = 0.87. It does not, by itself, answer the questions an operator actually has to answer next:

    • Is this isolated, or are 96 drives of the same model triggering at once?
    • What telemetry should accompany the decision without shipping 22.6 MB?
    • Should 96 remediation missions run independently, or is this one systemic incident?
    • What happens when a migration succeeds on 73 drives, fails on 11, and becomes unsafe on 12?
    • Which of those actions require approval? How is the whole operation replayed and audited afterward?

    Those are Context, Governance, and Mission questions, and this benchmark already answers two of them with hard numbers that no better predictor erases: reference-first context moves 95.3% fewer bytes, and the Governance Plane turns 96 individually-permissible actions into one fleet-level REQUIRE_REVIEW. That is also the cleaner story for accelerated hardware: put the best perception or prediction model on the device (a Jetson running Holoscan, a Metropolis fleet) and let ReDevOps be the runtime that turns those signals into context-efficient, governed, multi-device missions. You're betting on the layer that decides and governs what happens because of any one failure algorithm.

    The whole benchmark reduces to a few lines worth carrying out of this article:

    Adaptive routing: an always-on model's decision quality at 7% of its specialist inferences. Data-quality routing: the specialist's detection at 3% of its calls, and gating corrupted evidence lifts failure precision 0.300 → 0.850. 58% fewer false alarms. 95.3% less context moved. 24 days of warning. And 96 individually permissible actions → one fleet-level REQUIRE_REVIEW.

    What this is, and where it stops

    This validates the runtime's fleet capabilities on real telemetry with real ground truth. Adaptive discovery routing, reference-first context, and cross-device governance all do measurably what the architecture says they do, and they do it on data no one curated to flatter us. The through-line is one job: Discovery optimizes the path from uncertainty to verified state, deciding which capabilities are worth invoking, which evidence is worth acquiring, and whether that evidence is trustworthy enough to act on in the first place.

    It is not a claim to have the world's best drive-failure predictor: we showed the opposite, that off-the-shelf models cluster with a simple rule, on purpose. The claim is narrower and more durable: that prediction is one interchangeable input to a runtime whose job is to contextualize, correlate, govern, and execute what should happen because of it. Swap in a better model tomorrow and every result here still holds.

    It is also not a literal Jetson deployment. We haven't put this on a device yet, and we're not going to imply we have. A minimal on-device runtime slice, then Jetson-class hardware, is the next phase, and, deliberately, it's gated behind exactly this kind of measured result. That's the discipline: prove the capability on real data, then earn the hardware step.

    The benchmark is reproducible. The discovery rule, the leakage-safe learned comparators (GradientBoosting, RandomForest, XGBoost), the adaptive-routing frontier, the context accounting, and the governance evaluation all run from a committed fixture (self-test, python -m fleet) and reproduce in full from Backblaze's public quarterly archive. As with everything on our benchmarks page, the dataset is used to test the runtime, not shipped inside it. Customers point the runtime at their own fleets.

    A fleet is the honest test of whether you've built a runtime or just wrapped a model. This one says: Discovery routes to the cheapest sufficient capability and can recursively establish whether its own evidence deserves to be trusted; Context addresses only what the decision needs; Governance reasons across devices as a whole; one runtime optimizing the path from uncertainty to safe execution, candid about the 19% of failures that nothing can see coming.

    Want ReDevOps in your Google Top Stories & AI results?Follow ReDevOps on Google