← all benchmarksIntegration page →
Benchmark · integrations · NVIDIA NeMo

NVIDIA NeMo Agent Toolkit × ReDevOps — evidence

The full, reproducible evidence behind the integration page. A real nvidia-nat 1.8.0 agent, run in-process with the Mission SDK and Closure Resolution, measured on SciFact + GDPR + a governed action. Every comparison is classified — this is conformance, not a scorecard.

Frozen environment
framework nvidia-nat 1.8.0 (NeMo Agent Toolkit) — real Function via WorkflowBuilder.ainvoke runtime Mission SDK 0.2.0a0 · Context Runtime 0.2.0 — one Python 3.12 env, in-process answer model gpt-4o-mini (OpenAI-compatible) — identical across native and +ReDevOps arms datasets SciFact (allenai/scifact) · GDPR cross-reference graph (GDPRtEXT) budget SciFact 300 tok · GDPR 8000 tok (matched; the gold closure fits — budget is not the variable) scoring closure recall vs gold spans · verdict accuracy vs human labels · exactly-once side-effect witness

Results — NV-A through NV-E

ExperimentNative roleReDevOps additionResultClassification
NV-A SciFactNeMo agent + evaluatorClosure Resolution0.800 → 0.867REDEVOPS_DELTA
NV-B GDPRNeMo agentstructural closure0.412 → 0.913REDEVOPS_DELTA
NV-C OptimizationNeMo optimizationMission/context optimizationsemantics invariantEXPECTED_IMPLEMENTATION_DIFFERENCE
NV-D Governed toolNeMo proposes actionauthority · replay0 unauthorized · 0 duplicatesREDEVOPS_DELTA
NV-E TelemetryNeMo telemetryMission-root nestingpreserved (nested)EXPECTED_IMPLEMENTATION_DIFFERENCE
NV-A · scientific evidence (SciFact)

A NeMo agent verifies claims (Supported / Refuted / NotEnoughInfo) given retrieved evidence. Same model, same budget, two context sources. Verification accuracy naive 0.800 → closure 0.867; evidence recall 0.841 → 0.877. Treating the two context views as a cheap ensemble (agree ⇒ resolve, disagree ⇒ REQUIRE_REVIEW) gives selective autonomy of 0.857 @ 93% coverage. SciFact is content-saturated — semantic retrieval already finds most evidence — so the gain is real but modest.

NV-B · regulatory structure (GDPR)

The same closure mechanism on a structural domain: editing a GDPR article requires every article it cross-references, and cross-referenced articles are not textually similar. Content-only closure recall 0.412; composed (content + cross-reference graph) 0.913; seed-only 0.000 → 0.913. Deterministic — no LLM calls — so this isolates the retrieval value ReDevOps adds beneath the agent.

NV-D · governed action

A NeMo agent proposes a consequential action; it runs only through a Mission. Missing grant ⇒ 0 unauthorized actions. Grant but no approval ⇒ parked at a human gate, 0 side effects. Approved ⇒ fires exactly once. Replay of the sealed run ⇒ 0 duplicate side effects (reconstructed from the event ledger, not re-executed). The result that matters most for production is behaviour, not an accuracy delta.

Reproduce

git clone https://github.com/redevops-io/redevops-integrations cd redevops-integrations/integrations/nvidia ../../.venv/bin/python conformance.py # NV-A..E + classification tally ../../.venv/bin/python examples/scifact_closure.py # NV-A (gpt-4o-mini, cached) ../../.venv/bin/python examples/gdpr_structural.py # NV-B (no LLM) ../../.venv/bin/python examples/governed_tool.py # NV-D (real NeMo proposer, no LLM in the gate)

Acceptance bundles

Each run writes a reproducibility record — framework version, model ids, dataset hashes, per-finding classification, and a content-addressed result digest.

results/nv_a_scifact_closure.json nat 1.8.0 · gpt-4o-mini · scifact result_digest sha256:7ffd849e9efa7ff7 results/nv_b_gdpr_structural.json nat 1.8.0 · gdpr result_digest sha256:2855f48afa9098d8 results/nv_d_governed_tool.json nat 1.8.0 · gpt-4o-mini result_digest sha256:05aaad5650a6cb77 adapter_commit 0503d23 · mission-sdk 0.2.0a0 · context-runtime 0.2.0
// conformance summary   NVIDIA suite over all findings: 10 REDEVOPS_DELTA · 1 EXPECTED_IMPLEMENTATION_DIFFERENCE · 0 BUG. The one non-delta is NV-E: NeMo's telemetry nests under the Mission causal node — semantics preserved, shape differs. That is expected, not a win and not a bug.
Limitations. One answer model, one budget per domain, small samples (SciFact n=30, GDPR 27 articles); the managed NeMo Platform (registration/deployment) is not exercised — only the local NAT toolkit runs, for real. The SciFact selective-autonomy ensemble is two context views, not independent judges. Reported straight; nothing averaged into a single headline.

// reproducible · one Python env · github.com/redevops-io/redevops-integrations/integrations/nvidia  ·  integration page  ·  architecture article

© 2026 NVIDIA, the NVIDIA logo, NeMo and Nemotron are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. ReDevOps is an NVIDIA Inception member. Independent integration, not an NVIDIA product.