Cerebras Deploys Trillion-Parameter AI Models

Cerebras has developed a method to run trillion-parameter AI models on their wafer-scale chips. A single Cerebras WSE-3T wafer contains 44 gigabytes of on-chip memory. To support a model like Kimi K3, which has approximately 3 trillion parameters, Cerebras needed to distribute the model across multiple wafers. Kimi K3 is a 3 trillion parameter open-source model that demonstrates frontier-level coding and agentic capabilities.

The model comprises 69 linear-attention layers and 24 dense layers. With this distribution, Cerebras can run the model at 1000-2000+ tokens per second. The wafer-scale architecture allows for a pipeline execution of the model, where each wafer acts as a stage in an assembly line. As tokens flow through the pipeline, different stages of the model process them simultaneously.

This parallel processing enables the handling of multiple requests concurrently, boosting the overall throughput of the system. The number of concurrent requests is determined by the on-chip memory allocated for the key-value cache, which varies depending on the model mapping. By optimizing the distribution of the model across wafers and executing it in a pipeline, Cerebras can support models that exceed the capacity of a single chip, enabling the deployment of trillion-parameter AI models efficiently.