← all benchmarksIntegration page →
Benchmark · integrations · PydanticAI

PydanticAI × ReDevOps — evidence

The full, reproducible evidence behind the integration page. A real pydantic-ai 2.36.0 typed agent, run in-process with the Mission SDK, Closure Resolution and governance, measured on typed-validity vs admissibility, a governed action, and SciFact. Every comparison is classified — this is conformance, not a scorecard.

Frozen environment
framework pydantic-ai 2.36.0 (pydantic-ai-slim[openai]) — real typed Agent via run_sync runtime Mission SDK 0.2.0a0 · Context Runtime 0.2.0 — one Python 3.12 env, in-process answer model gpt-4o-mini (OpenAI-compatible) — identical across native and +ReDevOps arms datasets SciFact (allenai/scifact) · a strategy-intent + admissibility fixture budget SciFact 300 tok (matched; the gold closure fits — budget is not the variable) scoring typed validity vs admissibility · verdict accuracy vs human labels · exactly-once side-effect witness

Results — PA-A through PA-E

ExperimentNative roleReDevOps additionResultClassification
PA-A Governed intenttyped Agent → intentauthority · approval · exactly-once0 unauthorized · exactly 1REDEVOPS_DELTA
PA-B Telemetrypydantic-graph spansMission-root nestingpreserved (nested)REDEVOPS_DELTA
PA-C Typed replaytyped stateevent-sourced replayhash identical · 0 duplicatesREDEVOPS_DELTA
PA-D Validity vs verificationschema validityevidence · authority · governance4 refused · 1 admittedREDEVOPS_DELTA
PA-E SciFact closuretyped verdict AgentClosure Resolution0.800 → 0.867REDEVOPS_DELTA
typed-boundarytyped outputcanonical contract at boundaryno intrusionEXPECTED_IMPLEMENTATION_DIFFERENCE
PA-D · typed validity is not verification (the flagship, deterministic)

Five outputs, every one a schema-valid StrategyRecommendation Pydantic accepts. The runtime refuses four of them for reasons schema validity cannot express: not supported by the retrieved evidence, cited evidence stale beyond the freshness bound, the mission holds no authority grant for the action, and the security is governance-denied. The fifth is admissible and passes both layers. No LLM — the mechanism is isolated and reproduces bit-for-bit.

PA-A / PA-C · verified intent → governed action + typed replay

A real pydantic-ai typed agent produces a strategy intent (rebalance TLT, conf 85). Wrapped in a Mission: missing grant ⇒ 0 unauthorized actions; granted but unapproved ⇒ parked at a human gate, 0 side effects; approved ⇒ fires exactly once. Replay of the sealed run reconstructs the typed state to an identical content hash and fires 0 duplicate side effects. A schema-valid intent is not authority to act, and a restart never re-acts.

PA-E · scientific evidence (SciFact)

The same typed verdict agent on naive vs Closure-Resolution context, equal model and budget. Verification accuracy naive 0.800 → closure 0.867; evidence recall 0.841 → 0.877; selective autonomy (the two views as a cheap ensemble, agree ⇒ resolve, disagree ⇒ REQUIRE_REVIEW) 0.889 @ 90% coverage. This is the same closure mechanism the NVIDIA slice's NV-A measures on a NeMo agent — the near-identical numbers across two different frameworks are the semantic-conformance point.

Reproduce

git clone https://github.com/redevops-io/redevops-integrations cd redevops-integrations/integrations/pydanticai ../../.venv/bin/python conformance.py # PA-A..E + classification tally ../../.venv/bin/python examples/typed_verification.py # PA-D (no LLM, deterministic) ../../.venv/bin/python examples/governed_strategy.py # PA-A/PA-C (real typed agent, cached) ../../.venv/bin/python examples/closure_verdict.py # PA-E (gpt-4o-mini, cached)

Acceptance bundles

Each run writes a reproducibility record — framework version, model ids, dataset hashes, per-finding classification, and a content-addressed result digest.

results/pa_a_governed_strategy.json pydantic-ai 2.36.0 · gpt-4o-mini result_digest sha256:11a09227977c34e1 results/pa_d_typed_verification.json pydantic-ai 2.36.0 · no LLM result_digest sha256:2a64d9daaeedee95 results/pa_e_closure_verdict.json pydantic-ai 2.36.0 · gpt-4o-mini · scifact result_digest sha256:faccc60194881b02 adapter_commit 0997d46 · mission-sdk 0.2.0a0 · context-runtime 0.2.0
// conformance summary   PydanticAI suite over all findings: 12 REDEVOPS_DELTA · 1 PARITY · 1 EXPECTED_IMPLEMENTATION_DIFFERENCE · 0 BUG. The PARITY is PA-D's one admissible case (both layers agree it may proceed); the EXPECTED difference is the typed-boundary projection (the Pydantic model stays the local type). Neither is a win nor a bug.
Limitations. One answer model, one budget, small samples (SciFact n=30); PA-D is a constructed admissibility fixture, not a live trading system — it isolates the orthogonality of validity and verification, and is deliberately deterministic. PydanticAI's own retry/streaming and pydantic-graph multi-node flows are not stress-tested here. Reported straight; nothing averaged into a single headline.

// reproducible · one Python env · github.com/redevops-io/redevops-integrations/integrations/pydanticai  ·  integration page

PydanticAI and Pydantic are trademarks of Pydantic Services Inc. Independent integration, not a Pydantic product or endorsement.