← all benchmarksIntegration page →
Benchmark · integrations · LlamaIndex

LlamaIndex × ReDevOps — evidence

The full, reproducible evidence behind the integration page. A real llama-index-core 0.14.24 retriever and query engine, run in-process with the Mission SDK and Closure Resolution, measured on code closure, a long-context query engine, and a governed action. Every comparison is classified — this is conformance, not a scorecard.

Evidence · controlled · internally reproduced 27 Oct 2026 on the pinned code: closure recall 0.320 to 0.693 and contract-caller 0.300 to 0.600, identical.

Frozen environment
framework llama-index-core 0.14.24 — real VectorStoreIndex retriever + query engine runtime Mission SDK 0.2.0a0 · Context Runtime 0.2.0 — one Python 3.12 env, in-process embedding bge-small-en (local, offline CPU) — wrapped as a LlamaIndex BaseEmbedding answer model gpt-4o-mini (query-engine synthesis, LI-B only) — identical across arms datasets agentic-os pinned tree (code closure) · LiveRAG needle haystack (long context) budget code closure 4000 tok (matched across arms; only the representation varies) scoring closure / contract-caller recall vs the frozen oracle · needle accuracy · exactly-once side-effect witness

Results — LI-A through LI-E

ExperimentNative roleReDevOps additionResultClassification
LI-A Code closureLlamaIndex retrievercompose w/ structural closure0.320 → 0.693REDEVOPS_DELTA
LI-A Contract-callerLlamaIndex retrievercompose w/ call graph0.300 → 0.600REDEVOPS_DELTA
LI-A Composes/degradesretriever preferred if sufficient0 regressions1/5 tasks retrieval alonePARITY
LI-B Long-contextquery engine synthesisevidence-support abstention6/6 abstained · 1.000 @ 50%REDEVOPS_DELTA
LI-C Representationretriever is a representationselect / composecomposed, not replacedEXPECTED_IMPLEMENTATION_DIFFERENCE
LI-D Governed actionretrieval informsauthority · replay0 unauthorized · 0 duplicatesREDEVOPS_DELTA
LI-E Telemetryretriever/embed spansMission-root nestingpreserved (nested)REDEVOPS_DELTA
LI-A · code closure just outside the retrieval boundary (the flagship, no LLM)

A real LlamaIndex VectorStoreIndex retriever over 1,138 agentic-os symbols recovers the seed and type/schema edges the change description resembles, but misses the contract callers and transitive callers a correct change must also touch — those are structurally linked, not textually similar. ReDevOps registers the retriever as one representation and composes it with the static call-graph closure (retriever ∪ structural traversal, same budget): closure recall 0.320 → 0.693, contract-caller recall 0.300 → 0.600, with 0 tasks regressed. On the 1/5 tasks where semantic retrieval already resolves the closure, composition simply preserves it — the retriever is preferred when it suffices.

LI-B · long-context query engine (real synthesis)

A real LlamaIndex query engine answers "what is the code for facility X?" over a LiveRAG haystack with a buried needle. When the retriever finds the needle, the answer is right (1.000); when it misses, synthesis is confidently wrong (0.000) — retrieval quality, not synthesis fluency, decides correctness. On a miss the query engine still answers confidently (4 of 6) with no signal; ReDevOps evidence-support verification abstains on all 6/6 unsupported answers, giving selective accuracy 1.000 @ 50% coverage.

LI-D · governed action

A retrieved-evidence change runs only through a Mission. Missing grant ⇒ 0 unauthorized actions. Grant but no approval ⇒ parked at a human gate, 0 side effects. Approved ⇒ fires exactly once. Replay of the sealed run ⇒ 0 duplicate side effects. Retrieval quality is orthogonal to whether the write is allowed to fire.

Reproduce

git clone https://github.com/redevops-io/redevops-integrations cd redevops-integrations/integrations/llamaindex ../../.venv/bin/python conformance.py # LI-A..E + classification tally ../../.venv/bin/python examples/code_closure.py # LI-A (no LLM; real retriever over agentic-os) ../../.venv/bin/python examples/longcontext_answer.py # LI-B (real query engine, gpt-4o-mini, cached) ../../.venv/bin/python examples/governed_retrieval.py # LI-D (retrieval informs, Mission governs)

Acceptance bundles

Each run writes a reproducibility record — framework version, model ids, dataset hashes, per-finding classification, and a content-addressed result digest.

results/li_a_code_closure.json llama-index 0.14.24 · bge-small · agentic-os result_digest sha256:7738c0ec3d193ad7 results/li_b_longcontext_answer.json llama-index 0.14.24 · gpt-4o-mini · liverag result_digest sha256:9f07516e1b097950 results/li_d_governed_retrieval.json llama-index 0.14.24 · bge-small result_digest sha256:c945ceb1615cb6fc adapter_commit f54b11e · mission-sdk 0.2.0a0 · context-runtime 0.2.0
// conformance summary   LlamaIndex suite over all findings: 10 REDEVOPS_DELTA · 1 PARITY · 1 EXPECTED_IMPLEMENTATION_DIFFERENCE · 0 BUG. The PARITY is the composes-never-degrades check; the EXPECTED difference is LI-C (the retriever is composed as a representation, not replaced). Neither is a win nor a bug.
Limitations. Five refactor tasks over one pinned repo; one embedding, one budget. The composed arm is structurally seeded (a different input than NL retrieval), so its match recall is high by construction and is not reported as a retrieval win — only closure and contract-caller recall are. LI-B uses a small haystack to isolate retrieval hit/miss, not window size, and n is small. Reported straight; nothing averaged into a single headline.

// reproducible · one Python env · github.com/redevops-io/redevops-integrations/integrations/llamaindex  ·  integration page  ·  architecture article

LlamaIndex is a trademark of LlamaIndex, Inc. Independent integration, not a LlamaIndex product or endorsement.