The Engineered Drought: Why Cheap AI Inference Is Being Kept Out of Your Reach

There has never been more demand for AI inference (running models) and it's growing far faster than training. Training is a one-time capital event for a handful of labs; inference is forever, and everyone's. You'd expect the market flooded with cheap, inference-first hardware. It isn't. That is a choice physics permits.
Three prices, no middle
The AI hardware market today offers exactly three menus:
- "Trade in your car for a GPU." A single workstation accelerator (an NVIDIA RTX PRO 6000 Blackwell, 96GB of GDDR7, now around $13,000 retail), enough to run a serious model at home.
- "Take out a second mortgage for a GPU server." A box with 4–8 data-center GPUs, each carrying 100+GB of precious HBM, lands between $150k and $400k.
- "Ransomware-level, multimillion-dollar AI rig." A single 42U NVIDIA Vera Rubin (VR200 NVL72) AI rack is estimated at $7.8–8.8 million (industry and financial analyses): one rack, seven different chips, sold as a single extortion-priced unit.
Notice what's missing: the cheap, boring, inference-only box that should sit under tier one, the TPU-style part that does one thing, serve tokens, at commodity prices. The market has a hole exactly where demand is highest, and someone keeps it empty on purpose.
The bottleneck is memory, and memory is rationed
The scarce resource in 2026 is High-Bandwidth Memory (HBM), and compute is not. Demand grew roughly 5× between 2023 and 2026; supply is growing 50–60% a year. Only three companies make it (SK Hynix, Samsung, Micron), and the process (stacking 8–12 dies with through-silicon vias) yields between 55% and 75%. New fabs take 18–24 months. HBM is sold out years in advance, and its makers have unprecedented pricing power.
When a resource is that constrained, whoever controls it steers it to the highest-margin product: the multimillion-dollar training rig, and the affordable inference card is starved. HBM flows upward and the cheap tier is starved on purpose. The scarcity is real; the allocation is strategic.
Watch the vendors keep the hole open
Every major vendor made a move this past year that, conveniently, prevents a cheap inference part from existing:
- NVIDIA spent ~$20B at the end of 2025 to absorb Groq, the LPU inference-chip company, then made the resulting Groq 3 LPU available only as a decode co-processor inside the Vera Rubin rack (256 LPUs per rack). The one dedicated inference chip they own, you can't buy alone. NVIDIA sells its Vera ARM CPU separately, but it's highly specialized; AWS Graviton is the same story (AWS reportedly weighing selling it to other hyperscalers). Neither is meant for self-hosted local inference.
- AMD capped the Radeon AI PRO R9700 at 32GB of GDDR6 with no peer-to-peer Infinity Fabric, deliberately keeping it from scaling out, so it can't cannibalize the Instinct MI350/MI355X (288GB HBM3e, ~$30k).
- Intel has the most honest cheap-inference play in the industry, Crescent Island (Xe3P), which sidesteps HBM entirely with up to 480GB of LPDDR5X at a "cost-optimized" price. And it conveniently slips to 2027 (sampling H2 2026), the one part designed to break the drought, a year late.
The pattern is margin discipline (no conspiracy meeting): nobody whose revenue depends on $30k cards wants a $1,400 card that's "good enough" for inference. So the good-enough card keeps not shipping.
The CPU side is rigged the same way
A fat CPU with lots of RAM bandwidth serves a model fine without a GPU, especially for MoE models at INT8/FP8/Q4. So watch hardware FP8 on CPUs:
- Current Intel Xeon (Granite Rapids) and AMD EPYC (Turin) parts do AVX-512 and INT8, but native hardware FP8 lands only in the next generation. Just-released silicon stops one step short of the format that would make CPU inference trivially cheap.
- The one datacenter CPU with hardware FP8 today, Arm's AGI CPU, is effectively pre-booked by Meta, OpenAI, Cerebras and Cloudflare before launch.
2027: the dam breaks
The drought ends when (a) memory supply catches up and (b) a credible cheap-inference vendor ships at volume. Both are pointed at 2027:
- Qualcomm is the dark horse. It already has Oryon ARM datacenter CPU cores (via Nuvia) and Hexagon NPUs, and is in advanced talks to buy Tenstorrent (Jim Keller's RISC-V AI-accelerator company) for $8–10B. Its AI250 card (near-memory compute, early 2027) and the Tenstorrent line target Nvidia's blind spot: affordable inference without an HBM tax.
- Tenstorrent already sells the Blackhole P150a today: 32GB of GDDR6, no HBM at all, $1,399, open-source software stack.
- China, cut off from HBM and high-end GPUs by export controls, has built the cheap tier out of necessity: Huawei Ascend (used by DeepSeek), Moore Threads, Biren, Cambricon. When the global memory squeeze eases, those cost-first parts are positioned to flood exactly the niche the Western incumbents left empty.
The argument for the drought is "memory is scarce, be patient." The argument against: current-gen servers already run real models today for a fraction of the price, and the only reason a purpose-built cheap box doesn't exist is that it would eat into someone's $30k card.
We didn't wait. Here's the receipt.
This is practice. redevops.io runs its own agent fleet, including sidekick, our coding-agent orchestrator, on local, commodity inference. The benchmark that made us write this:
DeepSeek-V4-Flash (Q4), a frontier mixture-of-experts model, served on CPU only, on a single AMD EPYC 9655P (96 cores, ~$5k) with DDR5 system RAM. No GPU. No HBM. It sustained ~14.6 tokens/sec of decode and ~69 tokens/sec of prefill at a 64k context, driving real coding tasks through sidekick's
oncpubranch.
Fifteen tokens a second on a commodity CPU: faster than you read, cents of electricity, and the data never leaves your box. One honest caveat: hitting ~69 tok/s prefill means populating all 12 memory channels with DDR5, itself expensive in 2026, so this is a striking capability demo (a frontier MoE on CPU, no HBM) more than a guarantee of the cheapest box. But "you need an HBM supercomputer to do AI" is marketing. For inference (most of what businesses actually need), the affordable path already works.
What to do about it
- Right-size the model for a small rig. A well-quantized MoE on a CPU with lots of RAM beats an idle GPU you financed for three years.
- Treat HBM as an optional luxury. LPDDR-based parts (Intel Crescent Island, Qualcomm AI200/250) and GDDR6 parts (Tenstorrent, R9700) are the value tier: buy the memory you need, and skip the highest-margin memory.
- Keep inference local and portable. Self-host, stay quantization- and vendor-agnostic, and you route around the drought entirely.
- Wait out 2027 with eyes open. The cheap tier is coming, from Qualcomm/Tenstorrent in the West and from China at volume. Don't sign a multimillion-dollar rig lease betting it won't.
The inference drought is engineered. You don't have to live in it.
redevops.io builds rAIlsApps (source-available (AGPL-3.0 with the Commons Clause)): self-hostable modules run by agents, on hardware you can actually afford. See it run.
