OpenAI’s Jalapeño chip is built for fast inference at scale, benchmarks show

At the Hot Chips conference on Tuesday, OpenAI unveiled Jalapeño, a new chip designed for rapid inference at scale. The company released initial benchmark results for the chip, which outperformed the current state-of-the-art inference processors in terms of tokens per user and throughput per kilowatt. Richard Ho, OpenAI's head of hardware, stated, "The bottom line is that the results show a very, very significant performance advance over state of the art."

The benchmark tests were conducted against an Nvidia Blackwell system, but by the time Jalapeño is fully deployed, the competitive landscape may have shifted. OpenAI estimated that Jalapeño would enter limited production by the end of 2026, with wider release in 2027. Developed jointly with Broadcom, OpenAI's own models played a role in the chip's development.

The company aims to create a multigenerational platform, integrating AI products, models, chips, and memory. This full-stack approach allowed OpenAI to tackle specific challenges in the inference process, particularly reducing delays in the prefill and communication phases. By minimizing data movement and communication delays, Jalapeño can keep model state, like the KV cache used during response generation, local while activating the optimal compute, memory, and networking resources for each inference phase.