The Context Wars, Part I: The Hidden Cost of Context

Everyone is obsessed with context window size.
Nobody asks what it costs.
A million-token context sounds impressive, but every token must be read, represented, stored and attended to before the model produces a single answer. Long-context reasoning is one of the most expensive operations modern AI systems perform.
The quadratic cost of attention
At the heart of every Transformer is the attention mechanism. During prefill, the model builds relationships between tokens before generating the first output token. In dense attention, this computation grows roughly quadratically with context length: doubling the prompt can quadruple the work.
So long prompts are surprisingly expensive. A 100,000-token document consumes dramatically more GPU time, memory bandwidth and latency than a short one, beyond the added tokens.
RAG: retrieval as a compute optimization
This is why Retrieval-Augmented Generation (RAG) became the industry's default. Instead of asking the model to read everything, RAG moves the search problem outside the model.
Documents are split into chunks, each converted into an embedding and stored in a vector database. When a user asks a question, only the most relevant chunks are retrieved and sent to the LLM.
The economics are compelling: reading 2,000 selected tokens is far cheaper than processing 100,000, replacing expensive GPU computation with fast vector search on cheap infrastructure. RAG is a compute optimization and an accuracy feature.
Key takeaway
RAG became popular because GPU time is expensive. The most elegant solution was not what made it popular.
When embeddings aren't enough
But reality is messier. Vector embeddings capture semantic similarity well; they are less reliable when the answer depends on exact identifiers, serial numbers, legal clauses, function names or relationships spread across multiple documents.
That's why production RAG systems rarely rely on embeddings alone. They combine dense vector search with sparse keyword search such as BM25, fuse results with Reciprocal Rank Fusion, rerank candidates with cross-encoders, and link small child chunks to larger parent documents. Some build knowledge graphs to preserve relationships that simple chunking destroys.
Modern RAG, once 'retrieve a few chunks and hope for the best,' is an increasingly sophisticated pipeline built to compensate for the limits of vector similarity. For a practical implementation, I've open-sourced a reference architecture combining hybrid retrieval, reranking, parent-child chunking and production-ready retrieval strategies.
GitHub: github.com/redevops-io/redevops-rag
The guess no retrieval engine can make
And yet one problem refuses to disappear: no retrieval engine knows exactly what the model will need before it starts reasoning.
The retrieval layer makes an educated guess. Sometimes that guess is perfect. Sometimes the missing paragraph is the only one that mattered.
Developers respond by retrieving more chunks, but more context increases cost, latency and the chance the model loses focus.
Two answers, starting from opposite ends
The industry has spent the last few years optimizing retrieval because hardware forced us to. But what happens when the model itself becomes capable of managing enormous contexts more intelligently?
That question is dividing the AI industry into radically different architectural directions. DeepSeek believes the answer is changing the mathematics of attention itself. OpenAI appears to be investing more heavily in orchestration, reasoning workflows and system-level context management.
Both are trying to solve the same problem. One begins outside the model, the other inside it.
And that's where the Context Wars begin.



