Closure Resolution: Retrieval Isn't Finished Until You Know the Context Is Complete
Editor's note, 26 October 2026: The end-to-end ContractNLI results in this article (0.511 for the truncated full contract, 0.700 for naive retrieval and 0.733 for closure resolution, on 90 cases) come from a run recorded in a private benchmark repository: 30 cases per label sampled from the ContractNLI dev split, a 400-token budget, and the same model (gpt-4o-mini) on every arm. The per-case outputs were not kept, so the figures cannot be independently reproduced or tested for significance today, and our public evidence card for them says not to cite them as evidence of improvement until that is fixed. The 0.033 difference between closure resolution and naive retrieval is 3 of 90 cases. For a difference that small, an exact paired test cannot reach p < 0.05 whatever the pairing: the best case, with no case going the other way, gives p = 0.25. Closure resolution scored lower than naive retrieval on Entailment (0.567 against 0.600), and the Contradiction gain (0.800 to 0.933) is 4 of 30 cases. The 17-case gap between naive retrieval and the truncated full contract is large enough that a paired test could find it significant, but none has been run. The HippoRAG comparison reports its own interval and test; the other figures do not. A rerun that keeps the per-case outputs and reports a paired analysis is planned, and this note will be updated with its result. The article is otherwise unchanged. Evidence card: cr_contractnli_90case.
Ask a retrieval system for "the clause about data retention" and it will find you a clause about data retention. Whether that clause is enough to safely change anything is a different question, and most systems never ask it. The clause defines a term used three sections away; it is narrowed by an exception with no cross-reference; it is overridden by a later provision that shares none of its vocabulary. Retrieve the matching passage and act, and you have edited a contract while blind to the parts that decide what the edit means.
This is the same failure we keep meeting in different clothing. Change a function signature and the callers whose contracts break are not textually similar to the function. Verify a scientific claim and the sentence that refutes it is not the sentence that states it. The answer is rarely in the representation you queried. The unit of relevance is the dependency closure: the complete set of things that must be considered together to resolve the task.
We have spent the last stretch turning that intuition into a measured capability we call Closure Resolution, and testing it hard enough to find where it breaks. This post is the honest account: what generalizes, what doesn't, and the one distinction that has to stay sharp or the whole result gets misread.
Where it sits in the Runtime
Closure Resolution is a capability composed from parts the Runtime already has. It is easiest to understand as four separate questions:
selection what evidence is even allowed to enter?
closure is the evidence set SUFFICIENT to resolve the task?
verification of the evidence we have, what can we trust?
governance may execution continue - or must a human look first?
Retrieval-augmented generation, as usually built, answers only the first. Closure Resolution adds the other three, and, importantly, it treats "we cannot establish completeness" as a first-class outcome.
The closure principle holds across five domains
We ran the same closure composition (retrieve content by meaning, promote structurally-linked units along the domain's own graph, fill a budget, measure how much of the gold closure we recovered) on five domains, with no per-domain tuning. Two are the original pair (code, legal); three are unseen and built from real primary sources (SciFact human-labeled claim/evidence; the GDPR cross-reference graph; the DrugBank/DDIMDL interaction graph).
domain seed-only -> composed content-only character
-------------------- --------------------- ------------ ---------------------------
code 0.36 -> 0.96 0.73 structural (graph closure)
legal (contracts) 0.26 -> 0.58 0.44 semantic (content-led, capped)
scientific citation 0.28 -> 0.87 0.86 semantic (content-recoverable)
financial regulation 0.00 -> 0.91 0.41 structural (cross-references)
clinical interaction 0.00 -> 1.00 0.17 structural (interaction graph)
Two things are true in every row. First, composed context beats the seed alone by a wide margin: the thing you matched on is almost never the thing you needed. Second, the composition that works is different in each domain, and it tracks the domain's character: semantic closures (legal, scientific) are recovered by meaning and are content-led; structural closures (code, regulation, pharmacology) need the graph, because cross-referenced articles and interacting drugs are simply not textually similar to their anchors (content alone gets 0.41 and 0.17 respectively).
One careful qualification
It would be easy to overclaim here, so we won't. The composition mechanism generalizes, but the operating point is capability-specific: it is a property of the capability read off the sweep in aggregate, and we have not shown the Runtime can infer it reliably from an individual task. When we tried to predict the right composition per task from cheap features, the features were weak predictors and per-task selection was no better than a fixed global setting. So the defensible architecture is: Closure Resolution is the generic capability; the representations and the composition policy are capability configuration. You do not need a universal controller that guesses an unknown problem's ontology from every query, and we have no evidence that you could build one from these signals.
(A second honest caveat, stated plainly on the benchmark page: for the structural domains the gold closure follows the same relation the structural leg traverses, so high structural recall there is expected: the informative signal is that content alone fails. And clinical interactions carry no descriptive text, so that domain is the pure-structural endpoint of the axis and does not provide a text-retrieval test.)
The legal ceiling is a genuine feature
Legal closure recall tops out around 0.58, and for a while that looked like the capability failing on a large semantic corpus. Pushed on, it tells a much more interesting story.
Budget is not the lever. Every gold closure in the corpus physically fits inside the token budget; recall is still 0.58. Increasing the budget does not help: it eventually raises recall only by including almost everything, and precision collapses. In fact F1 is highest at the smallest budget and falls monotonically as you add room.
And smarter retrieval barely moves it. We tried two ways to chase the residual (the members the content-led composition misses):
approach legal closure recall
---------------------------------------------- --------------------
content-led composition (baseline) 0.582
+ seed-anchored relation probes (Exp 3) 0.601
+ evidence-gated LLM-proposed edges (Exp 4) 0.602
Relation-targeted retrieval (searching for the missing exception or condition) recovers about 15% of what the baseline missed, for +0.019. Letting a strong model (Qwen-27B) propose specific relational edges, admitted only when it can cite a quote that actually appears in the target provision, reaches +0.020. Both plateau in the same place.
The reason is the important part. A large fraction of a human legal closure is joined by interpretation, with no quotable relationship. The reviewer includes a provision because, reading the whole contract, it bears on the seed, and no sentence you could retrieve says so. That portion is not recoverable by evidence-native retrieval at all, and fabricating an edge to reach it would be worse than missing it. The accuracy ceiling here is partly a property of the task itself.
Which is exactly why the interesting behavior is "know which part you can act on, and route the rest."
Three accuracy numbers: keep them apart
Here is the distinction that has to stay sharp. There are now three different accuracy concepts in this work, and collapsing them into one number would be the one thing that undermines an otherwise clean result:
1. closure / retrieval recall did we FETCH the right context?
(scientific 0.867, legal 0.582)
2. verification accuracy of the fetched candidates, are the keep/drop
decisions correct vs human labels?
3. selective autonomous accuracy accuracy on the decisions the system ACTS on
at a stated coverage itself, with the remainder routed to review
A number like "95.7%" is meaningless until you say which of these it is. It is not end-to-end closure recovery. Stated defensibly, on the scientific domain:
On SciFact verification decisions, a stronger panel achieves 91.0% accuracy at full coverage and 95.7% at 80% autonomous coverage, with the remaining 20% explicitly routed to review.
Governance: resolve what the evidence supports, abstain when it doesn't
The verification layer is an ensemble of independent judges. Disagreement is carried as a signal, and it drives a calibrated decision about what to resolve autonomously versus what to escalate.
On legal, with a deliberately mixed panel, a learned calibrator (using disagreement plus retrieval and structural support) lifts autonomous accuracy from 0.696 at full coverage to 0.873 at 80% coverage and 0.926 at 60%, routing the least-confident decisions to review. It clearly beats a crude vote-margin rule, because a weak, noisy panel needs the extra signal.
We then attacked the real weakness (the legal panel had a judge that was below chance) by running a stronger panel (gpt-4o, gpt-4o-mini, Qwen-27B) on the new scientific domain against human labels:
full coverage @ 80% coverage
------------------------ ------------- --------------
legal (mixed panel) 0.696 0.873
scientific (strong panel) 0.910 0.957
Two findings, both worth keeping. The governance behavior generalizes: calibrated abstention lifts accuracy on an independent semantic domain too, the same shape at a higher operating point. And a new nuance: with the strong panel the learned calibrator only ties a simple disagreement rule (0.957 vs 0.952), whereas on the weak legal panel it dominated. A strong, accurate panel makes raw disagreement already well-calibrated; the learned features earn their keep mainly when the panel is weak. The practical recipe is therefore strengthen the panel first, then calibrate.
That is a far better product claim than pretending retrieval reaches 100% everywhere:
Resolve what the evidence supports; detect when the closure is incomplete or interpretation-dependent; abstain on partial context.
For legal, medical, financial and large engineering workloads, "the system knew not to proceed" is worth more than two more points of recall.
Does it change the answer? The end-to-end test
Higher recall is a means. The question a customer actually asks is whether the agent gets the task right more often. So we ran the whole pipeline end to end on ContractNLI, the real task of deciding whether a contract Entails, Contradicts, or does Not Mention a hypothesis, with human gold labels. Three context arms, the same model and the same token budget on every one, and the closure arm anchored on a derived seed (the top retrieved provision, gold evidence excluded) so nothing leaks:
context arm (equal budget, same model) NLI task accuracy
---------------------------------------------- -----------------
full contract, truncated to budget 0.511
naive semantic retrieval (top-k by similarity) 0.700
closure resolution (structural closure + content) 0.733
Closure resolution wins at equal budget: it is which tokens get included. And the per-label breakdown shows the mechanism cleanly, exactly where the theory says it should be:
gold label naive closure
------------- ----- -------
Entailment 0.600 0.567 (slightly worse - structural noise on easy cases)
Contradiction 0.800 0.933 (+0.133 - the carve-out that flips the answer)
NotMentioned 0.700 0.700 (a wash)
The entire gain comes from Contradiction: the cases where the answer hinges on an exception or carve-out that is not textually similar to the hypothesis, the exact thing similarity retrieval misses and structural closure recovers. On easy Entailment cases the closure arm is marginally worse, because naive retrieval already had the decisive provision and structural inclusion just adds noise at a fixed budget. Dependency-complete context helps precisely where a dependency decides the answer. (Stuffing the whole contract head is the worst option of all: full-context is not a substitute for the right context.)
Then the governance half. An ensemble (gpt-4o-mini, gpt-4o, Qwen-27B) answers the same NLI on the closure context; the case is resolved autonomously only when the judges agree, otherwise it is routed to review:
autonomous (unanimous ensemble) 0.817 accuracy @ 67% coverage
the routed third, if forced 0.433 accuracy (near chance)
Abstaining on disagreement lifts autonomous accuracy to 0.817 at 67% coverage, and the routed third is exactly where the ensemble is unreliable: near chance if forced. The errors migrate into REQUIRE_REVIEW and away from wrong autonomous decisions. That is the whole claim, earned:
Dependency-complete context makes the agent complete the task correctly more often, and when completeness cannot be established, the Runtime abstains on partial context.
We are also, deliberately, not chasing a sixth domain or two more points of legal recall. Five domains establish breadth; two independently labeled semantic domains stress the governance behavior; and the end-to-end result now connects retrieval to outcome. The open frontier is productization and harder task families: multi-step missions where an incomplete closure should halt a plan as well as an answer.
Is the ceiling just our retriever? We checked: it isn't
An obvious objection: maybe the legal tail is recoverable and we simply used a weak retriever. So we ran the whole end-to-end benchmark again with HippoRAG (a graph-native, entity-and-relation retriever built for exactly the multi-hop "connect the distant pieces" retrieval this problem looks like) swapped in as the retrieval representation. Same 90 cases, same derived seed, same answer model, same budget, same governance panel; only the retriever changed, and no gold ever entered its index.
It did not help. On this workload HippoRAG recovers less of the gold closure than the tuned dense-hybrid (0.41 vs 0.53), so the downstream numbers are flat:
arm NLI task accuracy
naive semantic 0.700
current Closure Resolution 0.733
HippoRAG-only 0.644
HippoRAG + Closure Resolution 0.733 (ties; Δ = +0.000)
oracle closure 0.822
The HippoRAG-backed closure ties the current one exactly: Δ = +0.000, 95% paired-bootstrap CI [-0.067, +0.056], McNemar p = 1.00. Entity-graph retrieval over short, dense legalese does not beat a good dense-hybrid plus the explicit legal cross-reference graph; the domain's structure is already captured, and the multi-hop entity reasoning HippoRAG is good at is not what this task needs. The oracle arm (0.822) confirms there is retrieval headroom, but HippoRAG does not capture it.
This is a null result, and we are keeping it. It is the useful kind: it says the remaining legal tail is interpretive, past graph recovery, the exact conclusion the retrieval-lever plateau pointed to, and it means the governance behavior is doing load-bearing work in its own right. HippoRAG stays a candidate representation for corpora where cross-document multi-hop entity reasoning genuinely is the bottleneck; this workload is not one of them. That is what "one representation provider beneath the capability" means in practice: you test a new one honestly, at equal budget, and you promote it only where it earns its place.
The shape of the thing
Selection determines what evidence enters. Closure determines whether the evidence set is sufficient. Verification determines what can be trusted. Governance determines whether execution may continue. Closure Resolution is the capability that spans those four, and the code call-graph that started all of this turns out to be just one representation provider beneath it, alongside regulatory cross-references, interaction graphs, and semantic evidence retrieval.
The experiments now give us an empirical reason for that abstraction, beyond a tidier name. Relevant evidence leaves retrieval open. It's finished when you can tell whether the evidence needed to resolve the task is complete, and know when it isn't.
