OpenAI's upcoming Jalapeño chip looks like it'll be an inference beast

OpenAI unveiled its upcoming Jalapeño AI accelerator chip at the Hot Chips semiconductor development conference in Stanford. This chip, developed in collaboration with Broadcom, is the first of a series of custom silicon from OpenAI designed for AI inference tasks. Compared to Nvidia's GPU systems, OpenAI claims Jalapeño will deliver higher throughput and lower latency when it becomes available later this year and reaches volume production in 2027.

Though Jalapeño won't replace OpenAI's existing hardware partners, it's designed with AI in mind. Memory bandwidth is crucial for inference, and Jalapeño appears to excel in this area. Early benchmarks show the chip delivering 1.5x to 1.9x more AI work at peak throughput and 1.7x to 3.6x lower end-to-end latency than competitors like GPT-OSS-120B, DeepSeek R1, and Kimi K2.5. For ultra-low-latency inference, Jalapeño is 2.1x to 4.1x faster.

Each Jalapeño system contains 128 accelerators, providing 1.7 exaFLOPS of 4-bit compute, 27.5 TB of HBM4 memory, and nearly 2 petabytes per second of memory bandwidth. Compared to AMD and Nvidia's latest rack systems, Jalapeño uses between 40 and 60 percent of the power. The system design is based on a rack scale architecture similar to Nvidia's NVL72 or AMD Helios.