The CPU is back: Rethinking the CPU-GPU split for LLM inference

For the past three years, graphics processing units (GPUs) have been the go-to choice for handling large language model (LLM) inference. Central processing units (CPUs) typically manage only a small portion of the total compute required per request, while GPUs handle the intensive calculations. However, the nature of modern LLM inference is changing, as tool calls, multistep reasoning, and orchestration across specialized smaller models are increasingly common.

This has led to a shift in the CPU-to-GPU ratio, from 1:8 in training workloads to 1:1 in some cases, and even higher in agentic deployments.

GPUs excel at executing the same operation on thousands of data elements simultaneously, thanks to their tens of thousands of cores. This parallelism makes them incredibly fast at dense matrix multiplications, which are central to the forward pass of a transformer during inference. They are optimized for floating-point operations per second (FLOPS), making them ideal for the massive mathematical throughput required for AI models.

CPUs, on the other hand, are optimized for sequential, conditional, and branching logic. They have direct access to main system memory and are well-suited for the orchestration layer that wraps any model. CPUs handle tasks such as tool dispatch, code execution, Python runtimes, sandboxes, input/output (I/O), and agent loop control flow. Their strength lies in executing a chain of diverse commands rapidly, making them ideal for parsing JSON, handling network I/O, and checking security permissions.

Traditionally, the inference stack has seen CPUs playing a supporting role to GPUs. When a request arrives at the API server, it is tokenized and scheduled by the CPU. The CPU then hands the request to the GPU for the forward pass, which dominates the compute budget. The GPU performs tasks like attention, feed-forward layers, and sampling the next token, with a brief CPU sync to retrieve the output and update the scheduler. The CPU then collects the output and returns it to the end user.

However, the rise of agentic AI is changing the landscape. Agentic AI involves integrated agents that issue multiple model calls, execute tool calls, and manage complex decision trees. In such scenarios, the CPU plays a more significant role, handling a larger proportion of the work, including parsing outputs, making API calls, collecting results, and feeding them back into the system. This puts the CPU in a more central position and potentially makes it a bottleneck in the overall process.

The industry is now reevaluating where compute should live, moving away from the traditional CPU-to-GPU ratio. Advancements in CPU inference are increasing global efficiency, making CPUs a more attractive choice for serving LLM inference. The future of LLM inference may see a more balanced division of labor between CPUs and GPUs, with CPUs playing a more prominent role in the orchestration and coordination aspects of the process.