Planner & EXPLAIN

8 slides · the served-vs-abstained inspection surface

PLANNER · EXPLAIN1/8

The window into every context decision

  • redevops.io/planner is the EXPLAIN surface for the Context Runtime's choices.
  • Every resolution — which engine, which model, which capability — is inspectable here.
  • See not just the answer but why this answer, and what the alternatives were.
  • Trust comes from being able to read the decision, not from taking it on faith.
PLANNER · EXPLAIN2/8

Cost-based planning, not a fixed pipeline

  • The planner selects retrieval method, model tier, budget, and verification per request.
  • It optimizes utility against cost rather than running every stage every time.
  • Greedy-by-utility today for speed; OR-Tools CP-SAT on the roadmap.
  • A cost model for LLM work — the query planner analogy, made real.
PLANNER · EXPLAIN3/8

Intent and candidate generation

  • Deterministic rules classify intent and risk before any model runs.
  • Restricted or sensitive data force-upgrades the handling bucket for safety.
  • A cross-product then generates concrete retrieval-method candidates.
  • Hybrid retrieval is the default; graph is chosen for multi-hop questions.
PLANNER · EXPLAIN4/8

Candidate ranking with grounded probability

  • Each candidate is ranked on quality, cost, and a calibrated P(relevant).
  • Isotonic calibration turns raw scores into probabilities you can threshold.
  • The chosen plan is the best utility-per-cost, not the highest raw score.
  • The full ranked list is visible — including the ones that lost.
learned quality per armcommunity0.64chosenvector0.57hybrid0.56graph0.55bm250.00
PLANNER · EXPLAIN5/8

Served vs abstained

  • When calibrated confidence is too low, the planner abstains rather than guessing.
  • EXPLAIN shows the abstain decision and the reason beside every served answer.
  • Abstention is auditable: you can see exactly where and why the system declined.
  • A grounded 'I don't know' is a first-class outcome, not a failure.
PLANNER · EXPLAIN6/8

Provenance as the record

  • Every decision carries resolver, representation, score, and alternatives.
  • The same provenance the runtime produces internally is what the surface renders.
  • No separate logging layer to drift out of sync with the real decision.
  • One EXPLAIN contract across knowledge, model, policy, and capability choices.
PLANNER · EXPLAIN7/8

Observability underneath

  • Standard traces via OpenTelemetry plus LLM-specific tracing via Langfuse.
  • EXPLAIN sits on top of real traces, not a hand-curated summary.
  • Inspect served versus abstained decisions and the reasons for each.
  • Full-fidelity view for debugging, tuning, and trust.
PLANNER · EXPLAIN8/8

Why it matters

  • Declare intent; the planner decides retrieval, budget, and verification automatically.
  • Every choice is inspectable and every decline is explained.
  • The self-optimizing decision layer improves as usage volume grows.
  • EXPLAIN is how a governed runtime earns trust — by showing its work.