Can We Drop 97% of Logs Without Missing the Next Failure?
Chain-aware filtering retains a fraction of log data—but detecting unfamiliar failures remains the hard part.
Most log pipelines filter by severity, keywords, or hand-written rules. Yet an INFO line can accompany a failed operation, while a WARN may be harmless. To understand what happened, we need to examine event chains—the messages associated with the same operation—not just individual lines.
We tested this approach on HDFS logs, training on the earlier half of the timeline and evaluating on the later half: 287,531 chains, including 6,668 anomalous chains.
Is log filtering and clustering still difficult?
The foundational problem has shifted from ingestion volume to meaning and cost.
-
Why log filtering is still hard: the "quiet failure" problem. Traditionally, filtering meant writing a regex or alert for static keywords like
ERRORor500. Today's microservices and LLM-powered applications often fail quietly, returning a200 OKor well-formed JSON, but with a completely incorrect output or an infinite agent loop. Filtering out the "noise" without accidentally dropping these subtle, high-impact anomalies remains incredibly challenging. Furthermore, commercial logging costs are skyrocketing, forcing companies to build complex edge-filtering pipelines just to keep their SaaS bills sustainable. -
Why log clustering is still hard: the context window tax. Clustering millions of raw log lines to find pattern variations (e.g., separating unique code errors from normal repetition) is computationally exhausting.
- The new paradigm (vector search + LLMs): The cutting edge has moved toward converting log messages into semantic text vector embeddings (using algorithms like MinHash) and applying sublinear nearest-neighbor search (like IVF-PQ) to cluster similar logs.
- The bottleneck: While smart sampling allows engineering teams to feed clustered log signatures into Large Language Models (LLMs) to get a natural-language root-cause summary, running this at gigabyte or terabyte scale in real-time is complex and easily blows past cost budgets if not tightly controlled.
What we measured
| Policy | Bytes retained | Anomaly recall |
|---|---|---|
| Chain-aware classifier + novel-pattern escalation, 99% target | 2.76% | 99.57% |
| Same policy, 99.9% target | 3.03% | 99.72% |
| Evidence lines only, 99% target | 0.82% | 99.57% |
| Template or any-WARN rules configured to meet the target | 100% | 100% |
The classifier kept less than 3% of the bytes while detecting over 99% of anomalous chains. The evidence-line result assumes selected lines are sufficient for diagnosis; we have not independently validated that assumption.
The stricter 99.9% recall target was missed: 19 anomalies went undetected. And the rule baselines' perfect recall came from retaining everything on this dataset—not proof that WARN-based filtering would recognize new kinds of failure.
Does more intelligence help?
We froze the classifier and tested retrieval only on unfamiliar-pattern chains it would discard. That review pool contained 27 of its 29 misses at the 99% operating point.
| Additional retrieval | Misses recovered | Extra bytes retained |
|---|---|---|
| Dense, with identifiers | 22 of 29 | +0.13 percentage points |
| SPLADE, with identifiers | 22 of 29 | +0.26 percentage points |
| BM25, with identifiers | 27 of 29 | +0.43 percentage points |
Retrieval improved the low-retention end of the tradeoff: dense retrieval recovered 20 misses for +0.11 percentage points, versus +0.27 from lowering the classifier threshold. But retrieval could not reach two misses with familiar patterns. An oracle threshold chosen using test labels recovered all 29 for +0.28 percentage points; that is not a deployable calibration result.
The experiment also exposed a calibration flaw: the calibration slice overlapped training data, and fitting had initially depended on input ordering. After fixing ordering, the Mac result reproduced exactly, but one threshold-boundary chain still flipped on Graviton. Freeze fitted artifacts, not just code.
A small Decider model did not earn a place: its calibration calls contained no remaining anomalies, so its incremental value could not be validated. Shadow-mode proposals to discard additional chains lost real anomalies. No suppression by retrieval or the Decider is permitted.
The next failure is the real test
We reproduced the classifier on an isolated ARM64 MicroVM using presigned S3 downloads and no VM credentials. The HDFS analysis took 30.6 seconds after a 1.1-second transfer of 39 MB. The retrieval and Decider experiments ran on the Mac, not Graviton; projected VM cost is not a verified AWS bill.
For now, we retain the simpler classifier and will recalibrate it on a disjoint chronological slice with enough anomalies. Then we will test on additional corpora and investigate whether retrieval, operational contracts, and dependency evidence can recognize genuinely novel failures, not merely historical anomalies near a decision threshold.
The objective isn't to classify yesterday's incidents more cheaply. It's to retain the evidence needed to recognize when an operation goes wrong—even when the failure has never been labeled before.
Research status: experimental. These results are HDFS-specific; 99.9% recall, novel-failure generalization, and independent diagnostic sufficiency remain unproven. Implementation and results: redevops-benchmarks draft PR #40.