All articles
    rag
    context-windows
    llm-inference
    attention
    retrieval
    graphrag
    deepseek
    openai
    ai-engineering

    The Context Wars, Part I: The Hidden Cost of Context

    June 29, 2026redevops.io
    The Context Wars, Part I: The Hidden Cost of Context

    Everyone is obsessed with context window size.

    Nobody asks what it costs.

    A million-token context sounds impressive, but every token must be read, represented, stored and attended to before the model produces a single answer. Long-context reasoning is one of the most expensive operations modern AI systems perform.

    The quadratic cost of attention

    The contest to fit information through the context window
    Scroll the diagram sideways to read itFull size

    At the heart of every Transformer is the attention mechanism. During prefill, the model builds relationships between tokens before generating the first output token. In dense attention, this computation grows roughly quadratically with context length: doubling the prompt can quadruple the work.

    Dense attention scales quadratically: doubling the prompt roughly quadruples the attention computation during prefill
    Scroll the diagram sideways to read itFull size

    So long prompts are surprisingly expensive. A 100,000-token document consumes dramatically more GPU time, memory bandwidth and latency than a short one, beyond the added tokens.

    RAG: retrieval as a compute optimization

    This is why Retrieval-Augmented Generation (RAG) became the industry's default. Instead of asking the model to read everything, RAG moves the search problem outside the model.

    Documents are split into chunks, each converted into an embedding and stored in a vector database. When a user asks a question, only the most relevant chunks are retrieved and sent to the LLM.

    The economics are compelling: reading 2,000 selected tokens is far cheaper than processing 100,000, replacing expensive GPU computation with fast vector search on cheap infrastructure. RAG is a compute optimization and an accuracy feature.

    Key takeaway

    RAG became popular because GPU time is expensive. The most elegant solution was not what made it popular.

    When embeddings aren't enough

    But reality is messier. Vector embeddings capture semantic similarity well; they are less reliable when the answer depends on exact identifiers, serial numbers, legal clauses, function names or relationships spread across multiple documents.

    That's why production RAG systems rarely rely on embeddings alone. They combine dense vector search with sparse keyword search such as BM25, fuse results with Reciprocal Rank Fusion, rerank candidates with cross-encoders, and link small child chunks to larger parent documents. Some build knowledge graphs to preserve relationships that simple chunking destroys.

    A production retrieval pipeline: hybrid dense + sparse search, Reciprocal Rank Fusion, cross-encoder reranking, parent-child chunk expansion, and a knowledge-graph layer
    Scroll the diagram sideways to read itFull size

    Modern RAG, once 'retrieve a few chunks and hope for the best,' is an increasingly sophisticated pipeline built to compensate for the limits of vector similarity. For a practical implementation, I've open-sourced a reference architecture combining hybrid retrieval, reranking, parent-child chunking and production-ready retrieval strategies.

    GitHub: github.com/redevops-io/redevops-rag

    The guess no retrieval engine can make

    And yet one problem refuses to disappear: no retrieval engine knows exactly what the model will need before it starts reasoning.

    The retrieval layer makes an educated guess. Sometimes that guess is perfect. Sometimes the missing paragraph is the only one that mattered.

    Retrieval is an educated guess made before the model reasons, and sometimes the one paragraph left behind is the only one that mattered
    Scroll the diagram sideways to read itFull size

    Developers respond by retrieving more chunks, but more context increases cost, latency and the chance the model loses focus.

    Two answers, starting from opposite ends

    The industry has spent the last few years optimizing retrieval because hardware forced us to. But what happens when the model itself becomes capable of managing enormous contexts more intelligently?

    That question is dividing the AI industry into radically different architectural directions. DeepSeek believes the answer is changing the mathematics of attention itself. OpenAI appears to be investing more heavily in orchestration, reasoning workflows and system-level context management.

    Two architectural directions converging on the same problem: DeepSeek rewriting the mathematics of attention, OpenAI investing in orchestration and system-level context management
    Scroll the diagram sideways to read itFull size

    Both are trying to solve the same problem. One begins outside the model, the other inside it.

    And that's where the Context Wars begin.

    Want ReDevOps in your Google Top Stories & AI results?Follow ReDevOps on Google