← redevops.io  /  Benchmarks  /  Voice synthetic corpus · no real PHI
Voice assistant · data-quality & decision control

The runtime, around a phone assistant

An AI phone assistant acts on a chain of uncertain evidence — a noisy audio stream, an ASR transcript, an inferred intent, extracted slots, an EHR lookup, a synthesized reply. Any link can be silently wrong while every downstream step still looks reasonable. This is the same Discovery-as-data-quality result as the fleet benchmark, in a second, agentic domain: before acting, a Discovery stage establishes whether the evidence can be trusted — cheap checks first, expensive verification only where the decision is in doubt.

Evidence · synthetic · internally reproduced 27 Oct 2026: 1,235 unsafe-if-ungated actions, 2 executed; out of sample 1,827 and 2; fairness disparity 0.034; regression suite 24/24; out of sample, 25 of 1,573 clean turns were gated unnecessarily.

The pipeline & the test

The call is a composition of Discovery stages; each returns a trust verdict into one DecisionGate. Reuses the fleet benchmark's modality-neutral Capability contract unchanged. Ground truth is a synthetic healthcare corpus — never real PHI — where every turn carries the correct gate and whether trusting it ungated would take an unsafe action.

telephony/session → DTMF → [DQ audio] → ASR → [DQ transcript] → [DQ intent+slots] → [DQ EHR/PM] → [DQ retrieval] → [DQ policy/PHI] → action → [DecisionGate: ALLOW · CONFIRM · REQUIRE_REVIEW · DENY] → TTS

Decision lift — unsafe actions prevented

Detecting bad data is table stakes; the result that matters is the decision. We run the failure-prone corpus with no data-quality control (trust everything) versus gated by adaptive Discovery, and count the unsafe actions that execute — wrong-patient lookups, unauthorized PHI disclosure, double bookings.

1,235 → 2
in-sample. Of 4,000 turns, 1,235 would take an unsafe action ungated. Adaptive DQ discovery holds all but 2 for CONFIRM / REQUIRE_REVIEW / DENY — at the specialist on 13% of turns, gate-exact 1.000.
1,827 → 2
out-of-sample. On a shifted, noisier population (new seed, more non-native speech), it still prevents all but 2 of 1,827 unsafe actions — the frozen thresholds generalise.

Detection frontier — the routing result, again

Cheap device-local validators (null / range / DTMF / slot / EHR freshness / PHI policy) resolve most turns; only consequential turns whose evidence isn't pristine escalate to the expensive specialist (2nd-pass alignment, patient reconciliation, groundedness).

strategy · catch turns that need gatingprecisionrecallspecialist calls
Cheap validators onlymisses the hidden defects (confident hallucination, wrong patient, injection)1.0000.7870.00×
Always specialistexpensive verification on every turn1.0001.0001.00×
Discovery — adaptiveescalate only the consequential-uncertain minority1.0000.9990.13×
Same frontier, second domainAdaptive reaches the always-on detector's recall (0.999 vs 1.000) at the specialist on 13% of turns — the decision-quality / resource frontier, now for agentic correctness rather than drive failure.
Holds out-of-sampleOn the shifted population the specialist fraction rises to 0.23× (noisier evidence, more escalation) and detection holds at 0.990 / 0.999 — the routing boundary isn't tuned to one distribution.

// 4,000 synthetic turns · gate oracle = expected_gate · 24-scenario regression suite 24/24 safe-side · reproduce: redevops-edge/voice · python -m voice

Fairness — and what out-of-sample honestly surfaced

A quality gate that "works" by silently re-prompting every non-native speaker has failed — more so in healthcare. Fairness is a blocking result, not a footnote: false-recovery (gating a genuinely clean turn) is measured for parity across accent groups.

0.000
in-sample disparity. False-recovery 0.000 native vs 0.000 non-native — no penalty for accented speech.
0.034
out-of-sample disparity. On the noisier population, non-native false-recovery rises to 0.034 — a small gap the in-sample run hid. Under the bar, but a real flag to widen accent handling before production. This is exactly why out-of-sample evaluation is run.
What is and isn't claimed. The corpus is synthetic — never real PHI. This benchmark does not establish "HIPAA compliance"; it tests HIPAA-relevant data-handling controls (PHI detected → authorized? → minimum-necessary? → permitted? → audited?) routed through the same DecisionGate. It measures the decision-control layer at the evidence level; the audio-signal validators against real speech (Common Voice trust votes, NISQA MOS) are the next layer. And because the ground truth is disclosed injection into a synthetic corpus, absolute detection precision is 1.000 by construction — the load-bearing result is the decision lift and its out-of-sample stability, not the detection precision.

// synthetic healthcare transactions, never real PHI · in-sample seed 7 + out-of-sample seed 23 (shifted distribution) · reproduce: redevops-edge/voice · python -m voice

// No real patient data is used or shipped — the corpus is generated synthetically for the benchmark. The runtime reuses the same Capability contract as the fleet benchmark, demonstrating one modality-neutral Discovery primitive across telemetry and agentic voice.