All articles
    closure-resolution
    context-runtime
    retrieval
    rag
    verification
    governance
    selective-prediction
    benchmarks
    legal-ai
    scientific-claims
    code-retrieval
    composition

    Closure Resolution: Retrieval Isn't Finished Until You Know the Context Is Complete

    August 31, 2026redevops.io
    Closure Resolution: Retrieval Isn't Finished Until You Know the Context Is Complete

    Editor's note, 26 October 2026: The end-to-end ContractNLI results in this article (0.511 for the truncated full contract, 0.700 for naive retrieval and 0.733 for closure resolution, on 90 cases) come from a run recorded in a private benchmark repository: 30 cases per label sampled from the ContractNLI dev split, a 400-token budget, and the same model (gpt-4o-mini) on every arm. The per-case outputs were not kept, so the figures cannot be independently reproduced or tested for significance today, and our public evidence card for them says not to cite them as evidence of improvement until that is fixed. The 0.033 difference between closure resolution and naive retrieval is 3 of 90 cases. For a difference that small, an exact paired test cannot reach p < 0.05 whatever the pairing: the best case, with no case going the other way, gives p = 0.25. Closure resolution scored lower than naive retrieval on Entailment (0.567 against 0.600), and the Contradiction gain (0.800 to 0.933) is 4 of 30 cases. The 17-case gap between naive retrieval and the truncated full contract is large enough that a paired test could find it significant, but none has been run. The HippoRAG comparison reports its own interval and test; the other figures do not. A rerun that keeps the per-case outputs and reports a paired analysis is planned, and this note will be updated with its result. The article is otherwise unchanged. Evidence card: cr_contractnli_90case.

    Ask a retrieval system for "the clause about data retention" and it will find you a clause about data retention. Whether that clause is enough to safely change anything is a different question, and most systems never ask it. The clause defines a term used three sections away; it is narrowed by an exception with no cross-reference; it is overridden by a later provision that shares none of its vocabulary. Retrieve the matching passage and act, and you have edited a contract while blind to the parts that decide what the edit means.

    This is the same failure we keep meeting in different clothing. Change a function signature and the callers whose contracts break are not textually similar to the function. Verify a scientific claim and the sentence that refutes it is not the sentence that states it. The answer is rarely in the representation you queried. The unit of relevance is the dependency closure: the complete set of things that must be considered together to resolve the task.

    We have spent the last stretch turning that intuition into a measured capability we call Closure Resolution, and testing it hard enough to find where it breaks. This post is the honest account: what generalizes, what doesn't, and the one distinction that has to stay sharp or the whole result gets misread.

    A seed node on the left connected to a small cluster of nodes, its dependency closure, with one member glowing amber to mark a dependency the system cannot confidently resolve.
    Scroll the diagram sideways to read itFull size

    Where it sits in the Runtime

    Closure Resolution is a capability composed from parts the Runtime already has. It is easiest to understand as four separate questions:

    selection      what evidence is even allowed to enter?
    closure        is the evidence set SUFFICIENT to resolve the task?
    verification   of the evidence we have, what can we trust?
    governance     may execution continue - or must a human look first?
    

    Retrieval-augmented generation, as usually built, answers only the first. Closure Resolution adds the other three, and, importantly, it treats "we cannot establish completeness" as a first-class outcome.

    The closure principle holds across five domains

    We ran the same closure composition (retrieve content by meaning, promote structurally-linked units along the domain's own graph, fill a budget, measure how much of the gold closure we recovered) on five domains, with no per-domain tuning. Two are the original pair (code, legal); three are unseen and built from real primary sources (SciFact human-labeled claim/evidence; the GDPR cross-reference graph; the DrugBank/DDIMDL interaction graph).

    domain                seed-only -> composed   content-only   character
    --------------------  ---------------------   ------------   ---------------------------
    code                    0.36 -> 0.96            0.73          structural (graph closure)
    legal (contracts)       0.26 -> 0.58            0.44          semantic (content-led, capped)
    scientific citation     0.28 -> 0.87            0.86          semantic (content-recoverable)
    financial regulation    0.00 -> 0.91            0.41          structural (cross-references)
    clinical interaction    0.00 -> 1.00            0.17          structural (interaction graph)
    
    Closure recall across five domains: content-only recall versus the recall added by structural composition. Scientific citation gains almost nothing from structure (0.86 to 0.87) and legal little (0.44 to 0.58); code, financial regulation and clinical interaction gain most (to 0.96, 0.91 and 1.00), because their closures are structural and content alone reaches only 0.73, 0.41 and 0.17.
    Scroll the diagram sideways to read itFull size

    Two things are true in every row. First, composed context beats the seed alone by a wide margin: the thing you matched on is almost never the thing you needed. Second, the composition that works is different in each domain, and it tracks the domain's character: semantic closures (legal, scientific) are recovered by meaning and are content-led; structural closures (code, regulation, pharmacology) need the graph, because cross-referenced articles and interacting drugs are simply not textually similar to their anchors (content alone gets 0.41 and 0.17 respectively).

    One careful qualification

    It would be easy to overclaim here, so we won't. The composition mechanism generalizes, but the operating point is capability-specific: it is a property of the capability read off the sweep in aggregate, and we have not shown the Runtime can infer it reliably from an individual task. When we tried to predict the right composition per task from cheap features, the features were weak predictors and per-task selection was no better than a fixed global setting. So the defensible architecture is: Closure Resolution is the generic capability; the representations and the composition policy are capability configuration. You do not need a universal controller that guesses an unknown problem's ontology from every query, and we have no evidence that you could build one from these signals.

    (A second honest caveat, stated plainly on the benchmark page: for the structural domains the gold closure follows the same relation the structural leg traverses, so high structural recall there is expected: the informative signal is that content alone fails. And clinical interactions carry no descriptive text, so that domain is the pure-structural endpoint of the axis and does not provide a text-retrieval test.)

    The legal ceiling is a genuine feature

    Legal closure recall tops out around 0.58, and for a while that looked like the capability failing on a large semantic corpus. Pushed on, it tells a much more interesting story.

    Budget is not the lever. Every gold closure in the corpus physically fits inside the token budget; recall is still 0.58. Increasing the budget does not help: it eventually raises recall only by including almost everything, and precision collapses. In fact F1 is highest at the smallest budget and falls monotonically as you add room.

    And smarter retrieval barely moves it. We tried two ways to chase the residual (the members the content-led composition misses):

    approach                                        legal closure recall
    ----------------------------------------------  --------------------
    content-led composition (baseline)                     0.582
    + seed-anchored relation probes (Exp 3)                0.601
    + evidence-gated LLM-proposed edges (Exp 4)            0.602
    

    Relation-targeted retrieval (searching for the missing exception or condition) recovers about 15% of what the baseline missed, for +0.019. Letting a strong model (Qwen-27B) propose specific relational edges, admitted only when it can cite a quote that actually appears in the target provision, reaches +0.020. Both plateau in the same place.

    The reason is the important part. A large fraction of a human legal closure is joined by interpretation, with no quotable relationship. The reviewer includes a provision because, reading the whole contract, it bears on the seed, and no sentence you could retrieve says so. That portion is not recoverable by evidence-native retrieval at all, and fabricating an edge to reach it would be worse than missing it. The accuracy ceiling here is partly a property of the task itself.

    Which is exactly why the interesting behavior is "know which part you can act on, and route the rest."

    Three accuracy numbers: keep them apart

    Here is the distinction that has to stay sharp. There are now three different accuracy concepts in this work, and collapsing them into one number would be the one thing that undermines an otherwise clean result:

    1. closure / retrieval recall     did we FETCH the right context?
                                       (scientific 0.867, legal 0.582)
    2. verification accuracy           of the fetched candidates, are the keep/drop
                                       decisions correct vs human labels?
    3. selective autonomous accuracy   accuracy on the decisions the system ACTS on
       at a stated coverage            itself, with the remainder routed to review
    

    A number like "95.7%" is meaningless until you say which of these it is. It is not end-to-end closure recovery. Stated defensibly, on the scientific domain:

    On SciFact verification decisions, a stronger panel achieves 91.0% accuracy at full coverage and 95.7% at 80% autonomous coverage, with the remaining 20% explicitly routed to review.

    Governance: resolve what the evidence supports, abstain when it doesn't

    The verification layer is an ensemble of independent judges. Disagreement is carried as a signal, and it drives a calibrated decision about what to resolve autonomously versus what to escalate.

    On legal, with a deliberately mixed panel, a learned calibrator (using disagreement plus retrieval and structural support) lifts autonomous accuracy from 0.696 at full coverage to 0.873 at 80% coverage and 0.926 at 60%, routing the least-confident decisions to review. It clearly beats a crude vote-margin rule, because a weak, noisy panel needs the extra signal.

    We then attacked the real weakness (the legal panel had a judge that was below chance) by running a stronger panel (gpt-4o, gpt-4o-mini, Qwen-27B) on the new scientific domain against human labels:

                              full coverage   @ 80% coverage
    ------------------------  -------------   --------------
    legal (mixed panel)           0.696           0.873
    scientific (strong panel)     0.910           0.957
    
    Selective autonomous accuracy versus coverage on legal and scientific. As coverage falls and more decisions are routed to review, accuracy rises: the legal mixed panel goes from 0.696 at full coverage to 0.873 at 80 percent; the scientific strong panel from 0.910 to 0.957. This is verification accuracy, separate from closure recall.
    Scroll the diagram sideways to read itFull size

    Two findings, both worth keeping. The governance behavior generalizes: calibrated abstention lifts accuracy on an independent semantic domain too, the same shape at a higher operating point. And a new nuance: with the strong panel the learned calibrator only ties a simple disagreement rule (0.957 vs 0.952), whereas on the weak legal panel it dominated. A strong, accurate panel makes raw disagreement already well-calibrated; the learned features earn their keep mainly when the panel is weak. The practical recipe is therefore strengthen the panel first, then calibrate.

    That is a far better product claim than pretending retrieval reaches 100% everywhere:

    Resolve what the evidence supports; detect when the closure is incomplete or interpretation-dependent; abstain on partial context.

    For legal, medical, financial and large engineering workloads, "the system knew not to proceed" is worth more than two more points of recall.

    Does it change the answer? The end-to-end test

    Higher recall is a means. The question a customer actually asks is whether the agent gets the task right more often. So we ran the whole pipeline end to end on ContractNLI, the real task of deciding whether a contract Entails, Contradicts, or does Not Mention a hypothesis, with human gold labels. Three context arms, the same model and the same token budget on every one, and the closure arm anchored on a derived seed (the top retrieved provision, gold evidence excluded) so nothing leaks:

    context arm (equal budget, same model)          NLI task accuracy
    ----------------------------------------------  -----------------
    full contract, truncated to budget                    0.511
    naive semantic retrieval (top-k by similarity)        0.700
    closure resolution (structural closure + content)     0.733
    

    Closure resolution wins at equal budget: it is which tokens get included. And the per-label breakdown shows the mechanism cleanly, exactly where the theory says it should be:

    gold label       naive   closure
    -------------     -----   -------
    Entailment        0.600    0.567   (slightly worse - structural noise on easy cases)
    Contradiction     0.800    0.933   (+0.133 - the carve-out that flips the answer)
    NotMentioned      0.700    0.700   (a wash)
    

    The entire gain comes from Contradiction: the cases where the answer hinges on an exception or carve-out that is not textually similar to the hypothesis, the exact thing similarity retrieval misses and structural closure recovers. On easy Entailment cases the closure arm is marginally worse, because naive retrieval already had the decisive provision and structural inclusion just adds noise at a fixed budget. Dependency-complete context helps precisely where a dependency decides the answer. (Stuffing the whole contract head is the worst option of all: full-context is not a substitute for the right context.)

    End-to-end ContractNLI task accuracy at equal budget: truncated full-context 0.511, naive retrieval 0.700, closure resolution 0.733; and the per-label breakdown showing the entire gain concentrated in Contradiction (0.800 to 0.933) where an exception flips the answer.
    Scroll the diagram sideways to read itFull size

    Then the governance half. An ensemble (gpt-4o-mini, gpt-4o, Qwen-27B) answers the same NLI on the closure context; the case is resolved autonomously only when the judges agree, otherwise it is routed to review:

    autonomous (unanimous ensemble)   0.817 accuracy  @ 67% coverage
    the routed third, if forced        0.433 accuracy  (near chance)
    

    Abstaining on disagreement lifts autonomous accuracy to 0.817 at 67% coverage, and the routed third is exactly where the ensemble is unreliable: near chance if forced. The errors migrate into REQUIRE_REVIEW and away from wrong autonomous decisions. That is the whole claim, earned:

    Dependency-complete context makes the agent complete the task correctly more often, and when completeness cannot be established, the Runtime abstains on partial context.

    We are also, deliberately, not chasing a sixth domain or two more points of legal recall. Five domains establish breadth; two independently labeled semantic domains stress the governance behavior; and the end-to-end result now connects retrieval to outcome. The open frontier is productization and harder task families: multi-step missions where an incomplete closure should halt a plan as well as an answer.

    Is the ceiling just our retriever? We checked: it isn't

    An obvious objection: maybe the legal tail is recoverable and we simply used a weak retriever. So we ran the whole end-to-end benchmark again with HippoRAG (a graph-native, entity-and-relation retriever built for exactly the multi-hop "connect the distant pieces" retrieval this problem looks like) swapped in as the retrieval representation. Same 90 cases, same derived seed, same answer model, same budget, same governance panel; only the retriever changed, and no gold ever entered its index.

    It did not help. On this workload HippoRAG recovers less of the gold closure than the tuned dense-hybrid (0.41 vs 0.53), so the downstream numbers are flat:

    arm                              NLI task accuracy
    naive semantic                        0.700
    current Closure Resolution            0.733
    HippoRAG-only                         0.644
    HippoRAG + Closure Resolution         0.733   (ties; Δ = +0.000)
    oracle closure                        0.822
    

    The HippoRAG-backed closure ties the current one exactly: Δ = +0.000, 95% paired-bootstrap CI [-0.067, +0.056], McNemar p = 1.00. Entity-graph retrieval over short, dense legalese does not beat a good dense-hybrid plus the explicit legal cross-reference graph; the domain's structure is already captured, and the multi-hop entity reasoning HippoRAG is good at is not what this task needs. The oracle arm (0.822) confirms there is retrieval headroom, but HippoRAG does not capture it.

    This is a null result, and we are keeping it. It is the useful kind: it says the remaining legal tail is interpretive, past graph recovery, the exact conclusion the retrieval-lever plateau pointed to, and it means the governance behavior is doing load-bearing work in its own right. HippoRAG stays a candidate representation for corpora where cross-document multi-hop entity reasoning genuinely is the bottleneck; this workload is not one of them. That is what "one representation provider beneath the capability" means in practice: you test a new one honestly, at equal budget, and you promote it only where it earns its place.

    The shape of the thing

    Selection determines what evidence enters. Closure determines whether the evidence set is sufficient. Verification determines what can be trusted. Governance determines whether execution may continue. Closure Resolution is the capability that spans those four, and the code call-graph that started all of this turns out to be just one representation provider beneath it, alongside regulatory cross-references, interaction graphs, and semantic evidence retrieval.

    The experiments now give us an empirical reason for that abstraction, beyond a tidier name. Relevant evidence leaves retrieval open. It's finished when you can tell whether the evidence needed to resolve the task is complete, and know when it isn't.

    Want ReDevOps in your Google Top Stories & AI results?Follow ReDevOps on Google