The honesty rule. A deterministic, rule-based control is only worth shipping if you can say exactly what it catches and what it misses. So we measure on public labeled data and report the misses as plainly as the hits — a claim only counts on data the detector was not tuned on.
The security plugin screens retrieved evidence for one rule: evidence may influence belief, but it may never grant itself authority. The obvious question is whether a deterministic, zero-inference rule catches real attacks — so we scored it on deepset/prompt-injections (CC-BY-4.0; 662 texts labeled injection or legitimate) rather than asserting.
| Detector (held-out test, 60 injection / 56 benign) | Precision | Recall | F1 | False-positive rate |
|---|---|---|---|---|
| always-flag | 0.517 | 1.000 | 0.682 | 1.000 |
keyword ignore (baseline) | 1.000 | 0.050 | 0.095 | 0.000 |
| open validators | 1.000 | 0.050 | 0.095 | 0.000 |
| + OWASP corpus (enterprise) | 1.000 | 0.250 | 0.400 | 0.000 |
The enterprise corpus lifts recall 5× over the open baseline — 0.05 → 0.25 — at 1.000 precision and a 0.000 false-positive rate on this held-out test set (on the train split it reaches 0.276 recall at a 0.006 false-positive rate).
Recall of 0.25 means this gate, alone, misses most injections — and that is the point, not a disappointment. The dataset is multilingual (English + German) and heavy on subtle social-engineering framing, while the corpus is English-technique-based; recall is bounded by language coverage, and the corpus is extensible per deployment. What the gate buys is specific: a zero-inference, fully explainable, replayable first gate that produced zero false positives on the held-out benign set and costs no tokens — run before, not instead of, a model-based classifier, so the expensive classifier and human reviewers spend attention on the hard three-quarters.
A control you can prove, replay, and reason about is worth more as a first gate than a black box that is occasionally more sensitive and never explainable.
Reproduce: python benchmarks/prompt_injection_bench.py in runtime-plugins-enterprise (dataset downloaded on run). Companion reading: Security and Telemetry Belong to the Runtime.
The same discipline extends to the other plugins as their benchmarks land — each measured on open data, each reporting misses as plainly as hits:
See it run. The telemetry & containment plane is live in the demo — the intrinsic-security showcase runs the real monitor over an exfiltration-shape scenario (decisions → correlation → containment), beside the permissions plane; and the geospatial resolver runs on real harvested parcels at geo.redevops.io.
Want to understand the implementation? Enterprise Plugins →