Cerebras Deploys Trillion-Parameter AI Models

Cerebras has developed a method to run trillion-parameter AI models on their wafer-scale chips. A single WSE-3T wafer offers 44 GB of on-chip memory, but these models require terabytes for their weights alone. To address this, Cerebras splits large frontier models across multiple WSE-3T wafers, enabling the execution of models with over a trillion parameters at speeds of 1000-2000+ tokens per second.

The open source Kimi K3 model, with approximately 3 trillion parameters, serves as an example. The model consists of 69 linear-attention layers and 24 dense layers, achieving impressive scores on Terminal-Bench 2.1 (88.3%) and BrowseComp (91.2%). To run Kimi K3, Cerebras calculates that 59 wafers are needed, with each wafer containing 16 GB of space dedicated to each layer.

The model is processed in a pipeline where each wafer runs a set of layers and passes activations to the next wafer via RDMA over Ethernet. This method ensures that the model weights stay on-chip, while activations move between wafers. This approach allows for full pipeline utilization, enabling the system to process multiple requests simultaneously rather than sequentially.

The number of concurrent requests is determined by the available on-chip memory and the size of the micro-batch. The throughput of the inference system is measured by total token throughput per unit of power, which can vary depending on the workload, configuration, and models being tested.