All articles
    context-runtime
    code-retrieval
    graph
    benchmarks
    redevops-rag
    agents
    mission-runtime

    Retrieving the Change, Not Just the Code

    August 30, 2026redevops.io
    Retrieving the Change, Not Just the Code

    Retrieval benchmarks usually ask one question: did you find the most relevant passage? For a coding agent, that is the wrong question.

    A code change is a closure: the definition you edit, plus every caller whose contract breaks, plus the type it flows through, plus the test that covers it. Find the function but miss one caller and the change doesn't compile. Retrieve the whole file and you've buried the model in 240,000 tokens of noise. "Find the relevant snippet" is necessary and nowhere near sufficient.

    So we built a benchmark that measures the thing that actually matters (does the change land?) and used it to test a specific architectural question: should code retrieval be a graph, a content retriever, or something else?

    The short answer: neither alone. Compose them. And the reason, past recall, is that the composition changes whether the coding task can be completed at all.

    The test

    Everything runs against a real repository (agentic-os, pinned at one commit, 1,138 indexed symbols, ~240k tokens). Every arm gets the same 4,000-token materialization budget. This is a benchmark about spending a fixed context budget well.

    Two things make it more than a recall benchmark:

    1. The closure oracle is declared before the patch is written, and frozen with a content hash. Each task states its seed, its required dependency closure by class (direct caller, transitive caller, type/schema, test…), and the subset of callers whose contract must change. No fitting the target to whatever a retriever happened to find.

    2. A retrieved context is fed to a real coding agent, which applies its edit into an isolated git worktree, and we run the tests. The agent sees only what its retrieval arm surfaced. A failing test can be handed back for another attempt, but no missing source is ever injected. So "did it resolve?" stays a clean function of what was retrieved.

    The agents are a local Qwen3.8-27B and, for cross-checking, Kimi-k2.7-code over the cloud API.

    Finding 1: content retrieval is genuinely good, and blind in a specific way

    Dense hybrid retrieval (embeddings + BM25 → reciprocal-rank fusion) is not a strawman. On a capability-metadata change it recovered the seed, the direct callers, and the type edge: closure recall 0.80 where plain lexical retrieval got 0.40.

    But two misses recurred, and both are structural:

    • Transitive callers: a caller several hops up the call chain that shares no words with the change.
    • Generic-bodied definitions: a function whose implementation ("provision credentials, invoke operator") looks nothing like the task that changes its signature.

    You cannot embed your way out of these. The signal is in the call graph, outside the text.

    Finding 2: closure recall decides whether the edit lands

    Before trusting any of this we had to rule out the boring explanation: maybe the model is just weak.

    It isn't. When we gave the agent the full oracle closure, it made the change and passed the tests, first try, on both Qwen and Kimi. When we gave it a partial closure that was missing the definition, it failed, on both models. A bigger model did not compensate for source that wasn't in the context.

    There was one honest detour worth reporting. Our first signature-change task was under-specified: it didn't say whether the new parameter had to stay backward-compatible. Both models independently made it required and broke existing callers. Same failure, two very different models: that's a task-specification bug, models and retrieval cleared. Once we stated the compatibility requirement, both models resolved the oracle context 2/2.

    The clean conclusion:

    When the task is well-specified, closure recall is the binding constraint on completing a code change. More model does not repair absent code.

    That is exactly the regime where retrieval architecture matters.

    Finding 3: graph traversal fixes content's blind spot, and has its own

    So we added a structural arm: seed on the symbol being changed, walk a bounded AST call graph (callers up to two hops, callees one), and materialize that closure under the same budget.

    On the transitive-parameter family (a change threaded down a three-level self-method chain), the graph recovered the callers content retrieval could not reach. Closure recall on the transitive-caller class went 0.0 → 1.0.

    And then the graph revealed its own blind spots. It resolves self.method() and module-level func() edges precisely, but it could not see:

    • a caller that constructs a type (Node(...), CapabilitySpec(...)), or
    • a call through a non-self receiver (self._executor_for(m).run()).

    So on the type-seeded and interface-change families, the graph underperformed content retrieval. The two representations fail on different classes of edge. That is the whole point.

    Finding 4: the union dominates, and the lift converts to resolution

    If they fail differently, compose them: take the graph closure first (the must-change members), then let dense hybrid fill the remaining budget. Closure recall, five families, one 4,000-token budget:

    familycontent (hybrid)graphgraph ∪ contentoracle
    capability-metadata0.800.800.801.00
    signature change0.670.331.001.00
    signature change (compat)0.670.331.001.00
    transitive threading0.501.001.001.00
    type-field addition1.001.001.001.00

    The union is at least as good as either representation on every family, and reaches full closure on four of five: on the signature change, the graph supplies the seed the embedder missed while the embedder supplies the callers the graph missed.

    Resolution is the headline, past recall. We ran the apply-and-test loop on the families where the arms disagree:

    familycontentgraphgraph ∪ contentoracle
    signature change (compat)0/20/22/22/2
    transitive threading0/22/22/22/2
    type-field addition2/22/22/22/2

    Read the first two rows carefully.

    • On the signature change, neither retriever alone gave the agent enough closure to complete the edit: both 0/2. Their union reached closure 1.00 and the agent resolved it 2/2, matching the oracle ceiling.
    • On the transitive change, content retrieval failed and the graph alone resolved it, by recovering callers the embedder could not reach.

    The retrieval improvement is the difference between a coding task that fails and one that passes its tests.

    Finding 5: extend the graph only where the data forces it

    The one family the union did not fully close was the capability-metadata change, and it pointed at a precise gap: the callers that had to change were the ones that construct the type. So we added exactly one edge kind (INSTANTIATES), resolving a bare Foo(...) to the unique class Foo, and nothing more ambiguous than that.

    It did what it should: the graph's recall on that family rose 0.40 → 0.80, it recovered the constructors on the type-field family at higher precision than content retrieval, and it added no false edges elsewhere.

    What remained unrecovered is instructive. The last missing caller was a method that had to be extended to read a brand-new field: a dependency that does not exist in the code yet. No structural graph over the current tree can see an edge the change is about to create. That is an honest limit of static retrieval, and naming it precisely is more useful than pretending the union is perfect.

    The discipline here is the point: retrieve less, structurally. Add an edge type only when a miss repeats across families, it extracts with high precision, and recovering it moves end-to-end resolution. One edge earned its place. The rest wait for evidence.

    Where this belongs in the stack

    It would be easy to over-build this into a "Graph Runtime." That's the wrong shape. The responsibilities split cleanly across the layers we already have:

    • ReDevOps RAG owns the retrieval primitive. The code graph is another indexing/retrieval plugin next to lexical, dense, and hybrid: parse structure, resolve symbol identity, store typed edges, traverse bounded caller/callee/instantiation edges from a seed, return ranked candidates with provenance. It is a provider of capability: it never decides that a task needs the graph.
    • Context Runtime owns the choice and the composition. Which representation (hybrid, graph, or the union) is worth buying for this step under this token and latency budget? Context Runtime selects it, allocates the budget across structural and content candidates, unions and materializes the final context, records the provenance in EXPLAIN, and learns from whether the change actually resolved. graph ∪ content is a Context Runtime strategy over RAG plugins.
    • Mission Runtime owns nothing graph-specific. It supplies evidence that can seed retrieval (the symbol a user named, a failing test, a stack trace) and consumes the resulting context. No if code_graph leaks into orchestration.

    That boundary is why the result generalizes. The same machinery that decides "structural graph vs. dense text for this code change" is the machinery that will later decide among a property graph, relational evidence, or geospatial context for other domains. Context Runtime decides which representation is worth buying; the representations live in RAG.

    Honest caveats

    This is a mechanism established. Five families is enough to show the complementarity and the resolution conversion; it is not enough to fit the full regression of resolution on closure across the space of changes. The graph arms are still seeded on the symbol to change: in production that seed has to come from the content retriever's top hit or from task evidence, and a traversal without a credible seed should abstain, with no expansion. The graph currently knows self-method, module-function, and unique-class-instantiation edges; receiver-resolved calls, overrides, and config/schema flows are future work, gated on the same evidence bar. And the cross-model check covered the specification finding alone. None of that changes the shape of the result. It sets the price of promoting it from a measured experiment to a default.

    The takeaway

    Finding, and completing, the change is the job, beyond the file. On real code, content retrieval and structural traversal fail on different edges; their union, under a fixed budget, recovers the change closure that neither reaches alone; and that recovery is what flips a coding task from failing its tests to passing them. So Context Runtime should treat structural graph traversal as one more evidence representation and compose it with content retrieval when the task calls for change-closure reasoning: bounded, seed-provenanced, and promoted on measured resolution, past retrieval recall.

    Retrieve less. Retrieve the closure.


    Does this hold beyond code? We ran the same experiment on real statutes, regulations, and SEC contracts, Retrieving the Change, Across Documents: the citation graph wins or ties on every case, and the one lever that lifts the ceiling is precision-controlled extraction, ahead of more edges.

    Want ReDevOps in your Google Top Stories & AI results?Follow ReDevOps on Google