All articles
    context-runtime
    retrieval
    graph
    legal-ai
    benchmarks
    redevops-rag
    citation-graph

    Retrieving the Change, Across Documents

    September 8, 2026redevops.io
    Retrieving the Change, Across Documents

    In Retrieving the Change, Not Just the Code we argued that retrieval benchmarks ask the wrong question. "Did you find the most relevant passage?" is necessary and nowhere near sufficient, because a change is a closure: the thing you edit, plus every unit whose meaning or validity depends on it. On code we showed that structural graph traversal recovers callers that content retrieval cannot reach, that their union recovers the closure neither reaches alone, and (the result that mattered) that the recovery flips a coding task from failing its tests to passing them.

    That left an obvious question. Code has an unusually clean dependency structure: a call graph you can parse. Is the "change is a closure" principle a fact about code, or a fact about documents that reference each other? If it is the latter, it should show up more starkly in the most reference-dense documents humans write: law.

    So we ran the same experiment on real statutes, regulations, and contracts. The answer is unambiguous, and it comes with a precise account of the one thing that actually limits it.

    The test

    Three corpora, each with a real, explicit reference graph and a deterministic closure oracle (no human labeling, no fitting the target to what a retriever happened to find):

    • Statutes: one title of the U.S. Code (Title 11, Bankruptcy) from the official USLM XML. Sections cite other sections in their operative text ("the automatic stay under section 362"); those are the edges. 280 sections, 295 edges.
    • Regulations → statutes: SEC rules (17 CFR Part 240) that incorporate the Securities Exchange Act (U.S. Code Title 15). This is the genuinely cross-document case: an edit to a statute propagating to every regulation that cites it. 618 cross-document edges.
    • Contracts: a real credit agreement pulled from SEC EDGAR (CONMED's 2021 Amended & Restated Credit Agreement), parsed into its internal section-citation graph. 130 sections, 93 edges. Plus its defined-term graph: §1.1 defines a term, provisions throughout use it, editing the definition affects every user.

    The change-closure of an edited provision t is deterministic: the provisions that transitively cite t (reverse-reachability over the reference edges). That set is the oracle. Every retrieval arm gets the same 4,000-token materialization budget; recall is how much of the closure it recovers within that budget. We pit the citation graph against BM25, dense embeddings, and a HippoRAG-style associative graph (similarity-derived PageRank, a graph built from similarity).

    Finding 1: the citation graph wins every case, in every domain

    domaincasescitation graphbest similaritydecisive (graph / content)
    statutes (US Code Title 11)680.5560.159 (dense)51 / 0
    regulations → statutes (17 CFR 240 → USC 15)210.4840.066 (dense)21 / 0
    contract (EDGAR credit agreement)470.3020.209 (assoc.)29 / 0
    defined-terms (same contract)541.0000.206 (BM25)52 / 0

    Content retrieval never wins a single case, in any domain. That is the exact inverse of what we saw on a single NDA (the ContractNLI workload), where topically-similar clauses largely were the related ones and BM25 nearly tied the graph. The difference is structural: within one short document, similarity is a decent proxy for dependence; across a real corpus of cross-referencing documents, it is not.

    And the gap widens exactly where you would predict, as the reference becomes less lexical:

    • Within a statute, content recall is 0.13. Across documents it collapses to 0.04. A rule that says "as required by 15 U.S.C. 78c" shares almost no surface text with §78c's actual language, so nothing lexical connects them.
    • Editing a definition, BM25 recovers only 21% of the provisions that use the term (a covenant that uses "Permitted Liens" reads nothing like the definition of "Permitted Liens") while the term-use edge recovers all of them.
    • The gap is largest at hubs, the high-citation provisions it is most important to get right: statutory hub recall is 0.33 for the graph versus 0.09 for BM25.

    Finding 2: an associative graph is not a citation graph

    The tempting shortcut is to say "so we need a graph" and reach for a similarity graph, HippoRAG-style: build edges from embedding similarity, run personalized PageRank. We included exactly that as a control. It does not help. On statutes it is worse than plain BM25 (0.031 vs 0.088 at hubs).

    The reason is the whole point. A similarity-derived graph cannot represent "clause A is bound to clause B because A cites B." Dress similarity up as a graph and it is still similarity: it still cannot see the dependency that shares no words. The win is the citation edges. (This is the same reason HippoRAG added ~0 on the single-document benchmark.)

    Finding 3: the ceiling is edge coverage, and only one lever lifts it

    If the citation graph is given the edges, the interesting question becomes: what happens when the edges are incomplete, as they always are when you extract them from messy real text? We tested this with edge-dropout (hide 30% of the real edges, a self-supervised oracle) and asked how to get them back. Two levers:

    Completing the graph by topology: fails. Every classical link predictor (common neighbours, Adamic-Adar, Jaccard, resource allocation, preferential attachment) recovers dropped edges at average precision below 0.02, and adding their predictions back lowers closure recall by injecting wrong edges. Legal citation graphs are not homophilous: a section cites another for a legal-dependency reason that has nothing to do with shared neighbours, so neighbourhood heuristics cannot invent the missing edge.

    Extracting the edges from the text: works. The lever is reading:

    • Deterministic extraction where the reference is stated but not in the canonical form: an "act-relative" resolver that maps "section 13(a) of the Act" to 15 U.S.C. 78m recovers +62% more edges. The defined-term graph is deterministic too, and takes recall from 0.21 to 1.0.
    • LLM extraction for the genuinely implicit references, the ones with no number and no name ("subject to the foregoing limitations," "such Loans," "except as otherwise provided herein"). An LLM that reads each section recovers dropped edges at 0.37, roughly 18× better than topological completion.

    Finding 4: precision over recall is what makes the LLM help

    The LLM lever has a trap that is worth stating plainly, because it inverts the usual intuition. A naive prompt ("list every section this one references") over-produces: 525 mostly-spurious edges, and adding them lowers closure recall, from 0.230 to 0.188. More edges made retrieval worse.

    Three precision controls fix it: a conservative prompt, a per-edge confidence threshold, and (the load-bearing one) grounding: the model must cite a verbatim trigger phrase that actually occurs in the source text, or the edge is dropped. That single filter kills hallucinated targets and, unlike a regex, still works on implicit, number-less references. With those controls and a stronger model:

    extractionedges keptdownstream closure recall
    naive dump (gpt-4o-mini)5250.188 (hurts)
    precision-gated (gpt-5-mini)310.261 (+0.03)
    precision-gated (Kimi k2.7)370.269 (+0.04)

    Precision control flips the LLM's contribution from net-negative to net-positive. Confidence and grounding cut the spurious edges from 525 to a handful, so the few that remain are real dependencies that help the closure instead of bloating it.

    The subtle part: the precise runs actually recover fewer of the hidden edges than the naive dump did (they trade recall for precision), and yet they help downstream more. At a fixed context budget, precision beats raw recall, because every noisy edge displaces a real closure member you could have retrieved instead. This is the discipline the code post argued for (retrieve less, structurally) now measured on the extraction side too.

    Where this belongs in the stack

    The same clean split as before holds, and gets sharper:

    • ReDevOps RAG owns the retrieval primitive. The citation graph is the cross-document structural sibling of hybrid content search: reverse-closure traversal over a dependency graph, composable with a content ranking under a token budget. It is now shipped as an optional, dependency-light module in redevops-rag (redevops_rag.citation_graph: ChangeClosureGraph for the traversal, PrecisionEdgeExtractor for the confidence-plus-grounding LLM edges). It is a provider of capability for a planner.
    • Context Runtime owns the choice and the composition, and here is the important refinement. On code, the structural graph was often the dominant representation and led the union. On legal prose, content dominates and structure contributes a smaller, selective lift, so the composition must be content-led: a fixed "graph first" order actually hurts there. Which representation deserves the budget, and in what order, is domain-dependent. That is precisely a Runtime decision, free of hardcoding in the retriever.

    The principle generalizes; the optimal composition policy does not. The invariant across code, statutes, regulations, and contracts was the change closure. Everything else (representation, budget, ordering, extraction model) is a decision a Runtime makes per task.

    Honest caveats

    The cross-document regulation track is modest in size (21 cases: statute sections cited by at least three rules in one CFR part), though the effect is unanimous. On the deterministic tracks the graph is given the edge set, so the finding is the size of the similarity gap (traversal still has a cost): the EDGAR and extraction tracks, where edges must be pulled from noisy text, are where the method earns its keep. And "closure" here is the reference/dependency graph, a review-closure proxy. Whether an amendment is legally correct is a separate axis that needs a qualified attorney; this work measures internal dependency completeness, and claims only that.

    The takeaway

    Finding and completing the change is the job. That was true on code, and it is true, more starkly, on the reference-dense documents of law: the citation graph recovers the affected provisions that similarity structurally cannot, a similarity-shaped graph does not stand in for it, the ceiling is edge coverage, and the lever that lifts it is disciplined extraction: precision first. So a Runtime should treat structural traversal as one more evidence representation, compose it with content according to which one dominates the task, and spend its extraction budget on precision over volume.

    Retrieve less. Retrieve the closure.

    Want ReDevOps in your Google Top Stories & AI results?Follow ReDevOps on Google