← all benchmarksenterprise plugins →
Enterprise benchmarks · plugins

Enterprise Benchmarks

Enterprise controls should be measurable, not asserted
These benchmarks measure the private security, telemetry, containment, and deployment plugins against public labeled datasets and recorded trajectories. Every result comes from a real offline run, and misses are reported alongside hits. The tests live in the private runtime-plugins-enterprise repo (py/benchmarks/) and are reproducible offline.
Securityprompt injection / authority manipulation
Telemetryplan-vs-observed divergence / exfiltration shape
Containmentresponse latency / no-override integrity
Supply chaindigest / provenance admission

The honesty rule. A deterministic, rule-based control is only worth shipping if you can say exactly what it catches and what it misses. So we measure on public labeled data and report the misses as plainly as the hits — a claim only counts on data the detector was not tuned on.

1 · Epistemic first gate — prompt injection

The security plugin screens retrieved evidence for one rule: evidence may influence belief, but it may never grant itself authority. The obvious question is whether a deterministic, zero-inference rule catches real attacks — so we scored it on deepset/prompt-injections (CC-BY-4.0; 662 texts labeled injection or legitimate) rather than asserting.

Method

Detector (held-out test, 60 injection / 56 benign)PrecisionRecallF1False-positive rate
always-flag0.5171.0000.6821.000
keyword ignore (baseline)1.0000.0500.0950.000
open validators1.0000.0500.0950.000
+ OWASP corpus (enterprise)1.0000.2500.4000.000

The enterprise corpus lifts recall 5× over the open baseline — 0.05 → 0.25 — at 1.000 precision and a 0.000 false-positive rate on this held-out test set (on the train split it reaches 0.276 recall at a 0.006 false-positive rate).

Read it honestly

Recall of 0.25 means this gate, alone, misses most injections — and that is the point, not a disappointment. The dataset is multilingual (English + German) and heavy on subtle social-engineering framing, while the corpus is English-technique-based; recall is bounded by language coverage, and the corpus is extensible per deployment. What the gate buys is specific: a zero-inference, fully explainable, replayable first gate that produced zero false positives on the held-out benign set and costs no tokens — run before, not instead of, a model-based classifier, so the expensive classifier and human reviewers spend attention on the hard three-quarters.

A control you can prove, replay, and reason about is worth more as a first gate than a black box that is occasionally more sensitive and never explainable.

Reproduce: python benchmarks/prompt_injection_bench.py in runtime-plugins-enterprise (dataset downloaded on run). Companion reading: Security and Telemetry Belong to the Runtime.

More enterprise benchmarks

The same discipline extends to the other plugins as their benchmarks land — each measured on open data, each reporting misses as plainly as hits:

Telemetry correlation — plan-vs-observed divergence and exfiltration-shape detection over recorded agent trajectories (building on the governance τ-bench corpus).
Containment — the disposition→containment state machine's response latency and no-override integrity under adversarial sequences.
Supply-chain admission — digest-pin / provenance gate coverage against a corpus of substituted and unsigned artifacts.

See it run. The telemetry & containment plane is live in the demo — the intrinsic-security showcase runs the real monitor over an exfiltration-shape scenario (decisions → correlation → containment), beside the permissions plane; and the geospatial resolver runs on real harvested parcels at geo.redevops.io.

Want to understand the implementation? Enterprise Plugins →

Open-core by design. ReDevOps publishes the runtime contracts, protocol shapes, reference runtimes, and reproducible public examples under AGPL — the inspectable foundation and adoption path. Enterprise Edition is where ReDevOps packages the production enforcement and integrations we maintain commercially for organizations operating consequential AI systems. Same contracts, stronger private enforcement behind the same seams.
ReDevOps enterprise benchmarks. Datasets are used only to benchmark the plugins (not redistributed, not shipped): deepset/prompt-injections CC-BY-4.0. Every number is from a real offline run; reproduction commands on each entry. See the enterprise plugins and the public benchmarks.