All articles
    ai-explainability
    retrieval-augmented-generation
    llm-interpretability
    trustworthy-ai

    Can You Explain Why Your AI Read That Document?

    July 10, 2026redevops.io
    Can You Explain Why Your AI Read That Document?

    Ask an AI system a question. Then ask the real question:

    Why did it read those documents?

    Most systems cannot answer. They can show citations and retrieved chunks. They cannot explain the decision that selected them.

    That is the problem.

    Citations are not explainability

    A citation tells you what the model used.

    It does not tell you:

    • why that source was retrieved
    • why another source was rejected
    • why this retrieval strategy won
    • whether the score was reliable
    • whether the source was allowed
    • whether the model should have abstained

    As retrieval gets more complex, this gets worse. Hybrid search. Graph retrieval. Reranking. Memory. MCP tools. Long context. Multiple data sources.

    More machinery means more decisions, and more to observe.

    The enterprise problem

    For internal systems, retrieval is relevance, and it is permissioning.

    A user may access:

    • this database
    • not that database
    • this table
    • not that column
    • this document
    • not that folder
    • these rows
    • not another user's rows

    If permissions are checked after retrieval, it is too late. The planner should only consider feasible plans, and policy should define the execution space before optimization begins.

    The remedy

    Treat access control as part of planning.

    A candidate plan is valid only if it satisfies:

    • user permissions
    • tenant isolation
    • data-source ACLs
    • row/column policies
    • budget limits
    • provider restrictions
    • verification requirements

    Then optimize only inside that allowed space.

    This is RBAC, and it is policy-constrained planning.

    The point

    Production AI needs to answer two questions:

    1. Why did the model read this?
    2. Was it allowed to?

    If the system cannot answer both, it is not ready for enterprise use.


    Reference implementation: https://github.com/redevops-io/context-runtime
    Benchmarks: https://redevops.io/benchmarks
    Whitepaper v2: https://redevops.io/whitepaper-v2


    redevops.io builds rAIlsApps (source-available (AGPL-3.0 with the Commons Clause)): self-hostable modules run by agents, on hardware you can actually afford. See it run.

    Want ReDevOps in your Google Top Stories & AI results?Follow ReDevOps on Google