All articles
    ai-runtime
    gpu-acceleration
    rapids
    cuvs
    cudf
    cuopt
    benchmark
    context-runtime
    evidence-native
    nvidia

    When the GPU Wins, and When It Doesn't

    August 22, 2026redevops.io
    When the GPU Wins, and When It Doesn't

    Part of the cross-infrastructure thread. Earlier we argued that compute should be a capability the runtime selects, free of assumptions baked into each app. This is the measurement behind that argument.

    There's a lazy way to put a GPU in your architecture and an honest one.

    The lazy way is to decide the system is now "GPU-accelerated," move everything you can onto the card, and quote the one benchmark where that looks good. The honest way is to treat the accelerator as a capability with a cost (one implementation of an operation among others) and to actually measure where it beats the CPU and where it doesn't. Sometimes the data is too small to overcome transfer overhead. Sometimes it's already sitting somewhere that makes moving it more expensive than processing it in place. Sometimes the CPU algorithm is just better.

    So before writing a line of "our runtime now runs on NVIDIA," we did the measurement. We took three operations from our evidence-native runtime, built a GPU implementation of each behind the same interface as the CPU one, and ran both on real Wikimedia revision history, with the CPU baseline on the same 96-core machine as the GPU, so the comparison is a real datacentre CPU against the accelerator beside it.

    One rule governed everything: correctness gates performance. A GPU result that isn't identical to the CPU reference doesn't get its time reported at all. The accelerator is allowed to change how fast an answer is produced, never what the answer is.

    Three operations, three very different verdicts.

    Retrieval: the clear win (cuVS)

    Retrieval is the operation people reach for a GPU for first, and here it earns it.

    The task: embed a query, find the nearest evidence, return it. We embedded 12,000 real revision texts with a real encoder, built the search index once, and then timed the thing a serving system actually pays per request: the query, the one-time build excluded. On the GPU that's NVIDIA cuVS; on the CPU, both an exact search (the ground truth) and a tuned hnswlib index.

    cuVS query latency stays flat near 3ms as the corpus grows to 200k vectors, while CPU exact search climbs from under a millisecond to 145ms (a 36x gap) at recall 0.96.
    Scroll the diagram sideways to read itFull size

    The shape is the whole story. CPU-exact search is a straight line up: every query rescans the whole corpus, so latency grows with it. The GPU query is flat at about 3 milliseconds no matter how big the corpus gets. They cross at around 12,000 vectors; by 200,000 the amortised GPU query is 36× faster than exact and about 3× faster than a tuned CPU index, and it's returning the same neighbours, recall 0.96 against the exact top-10. Identity preserved, latency crushed.

    That flat line is why retrieval belongs on the accelerator once a corpus is real-world sized. Below ~12k vectors, though, the CPU wins outright: the GPU's transfer-and-launch floor isn't worth paying for a handful of documents. The runtime should pick per corpus size, and now it has the number to pick on.

    Evidence processing: a win, but only at scale (cuDF)

    The second operation is the one our incremental discovery does constantly: take a body of evidence with a long revision history and compute its current state (filter by time, keep the latest revision per entity, correlate identical content). It's a dataframe reduction, and cuDF runs the exact same code as pandas on the GPU.

    cuDF versus pandas on the change-set computation: the CPU wins below 200k rows, the lines cross near a million, and the GPU reaches 5.7x faster at 2 million rows.
    Scroll the diagram sideways to read itFull size

    Here the honest answer is "it depends, and mostly on size." On the real strategywiki corpus (tens of thousands of revisions), the CPU wins: the operation finishes in under two milliseconds and the GPU's fixed ~3ms overhead is pure loss. The lines don't cross until a few hundred thousand rows, and the GPU only pulls decisively ahead past a million, hitting 5.7× at two million rows.

    The reading is that a single wiki's history is below the crossover, and a fleet of them (or a long-horizon evidence store that never forgets) is above it. Same operation, same code; the right backend flips with the size of the history. And at every size the change-set the GPU computed was byte-for-byte identical to the CPU's, with our incremental-equals-full-rescan invariant holding throughout. Faster, never different.

    Planning: the win the GPU never gets (cuOpt)

    The third operation is where accelerator honesty gets uncomfortable, and where it matters most.

    Assembling context is an optimisation: choose which evidence chunks to include to maximise relevance without blowing the token budget (a knapsack problem). cuOpt is NVIDIA's GPU solver for exactly this class of problem, so it's the natural thing to try. We put it head-to-head with the plain CPU dynamic program.

    cuOpt matches the exact DP optimum on every instance but runs 9 to 1000 times slower (hundreds of milliseconds against a sub-millisecond DP) so the GPU never wins and the CPU optimizer is kept.
    Scroll the diagram sideways to read itFull size

    cuOpt is correct: it found the exact same optimum as the dynamic program on all ten instances. It's also, for this problem, the wrong tool. The CPU DP finishes in well under a millisecond; cuOpt spends 100 milliseconds to a second of fixed setup and solve time to reach the same answer. That's the GPU being 9× to 1000× slower, and there is no crossover in any realistic context-assembly size. The runtime's correct decision here is to not use the accelerator and keep the CPU optimizer.

    This is the result we're most glad we measured, because it's the one the lazy approach gets wrong. A solver being GPU-accelerated and impressive on huge industrial problems tells you nothing about whether it beats a tight little DP on a knapsack of a few hundred items. It doesn't. (cuOpt's real home is large multi-constraint integer programs where no efficient DP exists, a different problem than the one context assembly poses.)

    Read the crossovers as hardware-specific

    One honest caveat, because it shapes every number above: this was a strong CPU against a memory-constrained GPU, and that pairing moves the crossovers.

    The CPU baseline is 96 cores with all 12 memory channels populated with DDR5-6400 (on the order of 600 GB/s of aggregate bandwidth and 392 GB of capacity), so the entire working set stays resident and the reductions never touch a page file. The accelerator is an RTX PRO 5000 Blackwell with 48 GB. Its on-package bandwidth is higher than the CPU's, but its capacity is a fraction of it, so once a dataframe or an index outgrows 48 GB, cuDF and cuVS have to stage data host→device across PCIe (an order of magnitude slower than either memory subsystem) and work in passes, while the CPU just keeps computing over resident memory.

    That asymmetry is a large part of why the crossovers land where they do. cuDF only pulls ahead past a million rows partly because below that the CPU's resident, high-bandwidth memory already wins outright and the GPU's transfer overhead isn't yet amortized, and cuOpt's fixed setup-and-transfer time is similarly punishing on the small knapsacks context assembly actually poses.

    So read the direction of each finding as robust (retrieval belongs on the accelerator, planning does not, evidence processing depends on scale) but the exact crossover size as a property of this hardware pairing. On a bigger-memory accelerator (an H100/B200 with 80–192 GB of HBM), or against a CPU with fewer memory channels, those crossovers would shift toward the GPU, some substantially. Which is the argument: the right place to compute an operation is a measurement, and it has to be taken on the hardware you actually deploy on, because the answer moves with the machine.

    The point is the choosing.

    Three streams of work crossing their thresholds at different distances: a metaphor for measured crossover, where the right processor changes with the scale of the operation.
    Scroll the diagram sideways to read itFull size

    Put the three verdicts next to each other and the actual finding appears:

    OperationAcceleratorVerdict
    RetrievalcuVSGPU wins above ~12k vectors, 36× at 200k
    Evidence processingcuDFGPU wins above ~1M rows, 5.7× at 2M; CPU wins below
    Context planningcuOptCPU wins, same optimum, 9–1000× faster

    Same runtime, three operations, three different right answers, and for two of them the answer even depends on how much data you have. No single "GPU-accelerated" label could be true across that table. What's true is narrower and more useful: for each operation there's a measured crossover, and the runtime should pick the backend on the right side of it.

    That's what "accelerator-aware, not GPU-dependent" means in practice. The CPU stays the reference and the fallback. The GPU is a capability with a cost (quality, latency, transfer, setup) that the planner weighs like any other, promoting it only where the measurement says it pays. And crucially, none of it touches the result: every accelerated answer here was identical to the CPU one, down to the byte or the objective value. The accelerator changes the economics of an operation, never its meaning, which is the only basis on which you'd let it into a governed system in the first place.

    We think that's the discipline GPU work in AI runtimes has mostly been missing. The question is "faster at what size, at what recall, at what cost, and is it still the same answer?" Those are the numbers that let a runtime decide where to compute, instead of a slogan that decides for it.


    Reproducibility: three arms (cuDF, cuVS, cuOpt) over the frozen strategywiki 20260801 history, real bge-small embeddings, run on an RTX PRO 5000 Blackwell 48 GB (RAPIDS 26.8) against a strong 96-core CPU baseline (all 12 memory channels populated with DDR5-6400, ~392 GB) on the same host, a pairing whose crossover points, as noted above, favour the CPU and would shift on a larger-memory accelerator. These are benchmark-side reference and accelerated implementations of the runtime's operations; wiring the winning backends behind the live capability interfaces is the next step. Full methodology, per-size tables and machine-readable results are in the benchmark's accelerator findings. rAIls, from ReDevOps, is a source-available Runtime Agentic Intelligence Stack for planning, context, execution and governance across production AI systems; the accelerator track is a private, benchmark-gated line of work and not part of the public release.

    Want ReDevOps in your Google Top Stories & AI results?Follow ReDevOps on Google