AI Chip Architectures

The 2018 International Symposium on Computer Architecture featured a Turing Lecture delivered by John Hennessy and David Patterson titled "A New Golden Age for Computer Architecture." In this era, post-Moore's Law, CPU single-threaded performance growth had slowed to 3% from the previous 52% annual growth in the 1980s. This marked a need for domain-specific architectures (DSAs), where Google's TPU v1 exemplified a successful application, achieving 29 times CPU throughput on neural-network inference with 80 times better energy efficiency.

As a result, various AI-focused architectures emerged, including GPUs (NVIDIA, AMD), systolic-array accelerators (TPU, Trainium), the Cerebras Wafer-Scale Engine, and the Groq LPU. NVIDIA emerged as the dominant player, with AMD following closely, backed by significant investments from tech giants like OpenAI and Meta. TPUs are utilized for training large models like Gemini, while Trainium chips power Anthropic's Claude, and Cerebras supports OpenAI's inference workloads. Additionally, Groq's LPU was acquired by NVIDIA for $20 billion.

AI compute predominantly consists of matrix multiplication operations. A transformer model, for instance, is a sequence of these multiplication steps, with each layer performing a large matrix-matrix multiply (GEMM) operation. During training, thousands of tokens are processed collectively through the weight matrix, while inference involves autoregressive decoding—token generation one at a time, conditioned on preceding tokens.

Inference systems improve performance through batching techniques such as continuous batching, speculative decoding, and multi-token prediction, all of which aim to optimize the utilization of matmul units.

Key architectural factors involve data storage locations, movement to compute units, compute unit design, and chip-to-chip communication at scale. Looking at NVIDIA GPUs, they feature massively parallel processing with thousands of threads orchestrated by a host CPU via CUDA, supporting a range of parallelisable workloads. The architecture comprises Streaming Multiprocessors, deep memory hierarchies, and scheduling silicon to manage thousands of threads concurrently.

Each Streaming Multiprocessor comprises 16k×32-bit register files, warp schedulers, dispatch units, special function units for transcendental functions, and private ports for Tensor Cores.

The Tensor Cores execute fused matrix multiply-accumulate operations on small matrix tiles, significantly enhancing AI workload performance. However, inference systems face challenges with data movement, as producing one token requires a full pass over all model weights and reading from the KV Cache for attention calculations. These bottlenecks lead to the memory wall—a limitation where compute scales exponentially, yet memory bandwidth does not.

Therefore, each architecture employs unique strategies for efficient data movement, addressing the memory wall challenge head-on.