Planner & EXPLAIN
8 slides · the served-vs-abstained inspection surface
PLANNER · EXPLAIN1/8
The window into every context decision
- redevops.io/planner is the EXPLAIN surface for the Context Runtime's choices.
- Every resolution — which engine, which model, which capability — is inspectable here.
- See not just the answer but why this answer, and what the alternatives were.
- Trust comes from being able to read the decision, not from taking it on faith.
PLANNER · EXPLAIN2/8
Cost-based planning, not a fixed pipeline
- The planner selects retrieval method, model tier, budget, and verification per request.
- It optimizes utility against cost rather than running every stage every time.
- Greedy-by-utility today for speed; OR-Tools CP-SAT on the roadmap.
- A cost model for LLM work — the query planner analogy, made real.
PLANNER · EXPLAIN3/8
Intent and candidate generation
- Deterministic rules classify intent and risk before any model runs.
- Restricted or sensitive data force-upgrades the handling bucket for safety.
- A cross-product then generates concrete retrieval-method candidates.
- Hybrid retrieval is the default; graph is chosen for multi-hop questions.
PLANNER · EXPLAIN4/8
Candidate ranking with grounded probability
- Each candidate is ranked on quality, cost, and a calibrated P(relevant).
- Isotonic calibration turns raw scores into probabilities you can threshold.
- The chosen plan is the best utility-per-cost, not the highest raw score.
- The full ranked list is visible — including the ones that lost.
PLANNER · EXPLAIN5/8
Served vs abstained
- When calibrated confidence is too low, the planner abstains rather than guessing.
- EXPLAIN shows the abstain decision and the reason beside every served answer.
- Abstention is auditable: you can see exactly where and why the system declined.
- A grounded 'I don't know' is a first-class outcome, not a failure.
PLANNER · EXPLAIN6/8
Provenance as the record
- Every decision carries resolver, representation, score, and alternatives.
- The same provenance the runtime produces internally is what the surface renders.
- No separate logging layer to drift out of sync with the real decision.
- One EXPLAIN contract across knowledge, model, policy, and capability choices.
PLANNER · EXPLAIN7/8
Observability underneath
- Standard traces via OpenTelemetry plus LLM-specific tracing via Langfuse.
- EXPLAIN sits on top of real traces, not a hand-curated summary.
- Inspect served versus abstained decisions and the reasons for each.
- Full-fidelity view for debugging, tuning, and trust.
PLANNER · EXPLAIN8/8
Why it matters
- Declare intent; the planner decides retrieval, budget, and verification automatically.
- Every choice is inspectable and every decline is explained.
- The self-optimizing decision layer improves as usage volume grows.
- EXPLAIN is how a governed runtime earns trust — by showing its work.