Cheap Inference Won't Solve AI

A few weeks ago I argued that AI has an inference cost problem.
OpenAI's reported Jalapeño chip pointed that way; Qualcomm's High Bandwidth Compute (HBC) architecture is a similar bet. Different designs, one objective: make AI inference dramatically cheaper and more efficient.
Inference is becoming its own market. Training frontier models and serving billions of requests are different workloads; optimizing for one doesn't optimize for the other.
Purpose-built inference hardware is good news. It will simply remove today's biggest bottleneck without solving AI.
Engineering has a habit of doing that
Cloud computing solved server provisioning. Bandwidth became the bottleneck. Faster networks solved bandwidth. Storage became the bottleneck.
Every major engineering breakthrough exposes the next constraint. Cheap inference will do the same.
One agent becomes ten. Then hundreds.
Today, running one AI agent continuously is expensive. Tomorrow it won't be. So instead of one agent, we'll run ten. Then fifty. Then hundreds of specialized agents on the same objective.
That's where the real challenge begins.
Reliability becomes the limiting factor
Even at an optimistic 95% success rate per task, chaining many dependent decisions drives the overall probability of success down. As workflows deepen and interconnect, reliability replaces inference cost as the limiting factor.
These are distributed systems questions.
Most agentic-AI discussion focuses on capabilities: Can an agent write code? Deploy Kubernetes? Review pull requests?
Production systems ask different questions:
- Can the same workflow produce the same outcome every time?
- Can failures be isolated?
- Can incorrect decisions be detected before they propagate?
- Can another agent independently verify the result?
- Can the entire execution be replayed and audited?
Agent swarms win because the software gets better
Agent swarms running 24/7 will succeed because the surrounding software gets dramatically better. They do not need models to become perfect.
Deterministic orchestration. Execution graphs. Checkpointing. Policy engines. Observability. Verification pipelines. Continuous validation.
We'll solve AI reliability the way we've solved every large distributed system for three decades: by designing systems that expect failure and recover.
When hardware stops being the bottleneck, software becomes the bottleneck
That's why Qualcomm's HBC announcement matters more than another benchmark. Like Jalapeño, it signals inference is steadily getting cheaper.
When hardware stops being the bottleneck, software becomes the bottleneck. And for the first time, AI engineering starts to look a lot like systems engineering.
redevops.io builds rAIlsApps (source-available (AGPL-3.0 with the Commons Clause)), self-hostable modules run by agents, on hardware you can actually afford.


