Nvidia and Cerebras Clash Over LLM Speed Claims at Hot Chips

Nvidia and Cerebras have entered a high-stakes performance battle over LLM inference speeds, sparking a debate over the gap between marketing benchmarks and production reality. The clash, centered on presentations at the Hot Chips conference, highlights a growing tension between raw latency and the economic requirements of serving multiple users.

Nvidia claims its Groq-3-based LPX racks can churn approximately 3,400 tokens per second when running the Gemma 4 31B model. Cerebras has countered with its next-generation CS-4 accelerators, claiming performance competitive with the LPX racks and asserting that its hardware is up to 30 times faster than traditional GPUs. These figures are part of a broader push toward "premium inference," where companies utilize SRAM-heavy architectures to push the boundaries of how quickly a model can generate text.

The Batch Size Controversy

Despite the impressive top-line numbers, reports indicate these figures are misleading because they are based on a batch size of one. In a production environment, serving a single user at a time is economically impractical for inference-as-a-service providers. Analysts describe these benchmarks as a "marketing gimmick," noting that while the speeds make inference feel instantaneous, they represent a "top speed on a race car" that customers will likely never see in a real-world deployment.

The Memory Bottleneck

This performance gap stems from the fundamental trade-off of SRAM-heavy architectures. While these chips offer extreme speed for a single stream of data, they possess significantly lower memory capacity than traditional GPUs. This creates a scaling wall: as more concurrent users are added, the memory required for the KV cache grows, quickly exhausting the available on-chip storage. To combat this, the industry is shifting toward heterogeneous architectures that combine the massive memory of GPUs with the fast decoding capabilities of specialized accelerators.

Market Implications

For the AI industry, the battle over tokens per second ignores the economic reality of throughput. Profitability in LLM deployment depends on high batch sizes—serving hundreds or thousands of users simultaneously on the same hardware. If the fastest chips cannot scale beyond a handful of users without massive hardware additions, their utility remains limited to niche, ultra-low-latency applications rather than general-purpose cloud AI.

What's Next

Attention now turns to how Nvidia integrates the technology from its $20 billion licensing and talent acquisition deal with Groq. While the acquisition of Groq's leadership and low-latency IP provides Nvidia with a significant architectural advantage, the industry will be watching to see if these systems can maintain their speed when subjected to the high-concurrency loads of actual production traffic.