Nvidia and Cerebras are selling performance their customers will (probably) never see

The Hot Chips conference in California saw Nvidia announce its new Groq-3-based LPX racks, with early tests showing the systems processing 3,400 tokens per second in Gemma 4 31B. This is four times faster than Cerebras' current offerings. However, Cerebras quickly responded by touting the performance of its next-gen CS-4 accelerators.

While these top-line performance figures may feel instantaneous compared to current chatbots, they are more of a marketing gimmick rather than a realistic expectation for customers. The reality is that the economics of premium or ultra-low latency inference are more complex, and the true measure of performance lies in how efficiently the system can scale that speed to handle a large number of users.

Both Nvidia and Cerebras are focusing on pushing the Pareto curve to the right, delivering massive bandwidth and extending performance, but the challenge lies in scaling that performance to accommodate a larger number of users. The key to making premium inference cost-effective lies in combining GPUs with Cerebras or Groq accelerators, as seen in partnerships between Nvidia, AMD, and AWS.

While top-line performance makes for great headlines, the real benchmark should focus on achieving a balance between interactivity, concurrency, throughput, and economics.