Why CPUs still matter in the age of AI agents

When discussing AI infrastructure, GPUs and TPUs are usually the first topics that come up. However, The New Stack recently spoke with Bhumik Patel of Arm and Mo Farhat of Google about the CPU, a chip that has largely gone unnoticed lately. Patel explains that CPUs are evolving into a more crucial role as AI shifts from chatbots to agents. He describes CPUs as "air traffic controllers" in this new era.

The workload has moved from simple answer-giving to actionable tasks. Agents perform actions by calling tools and creating environments to execute code. While large language models largely use accelerators, CPUs are essential for orchestration, data preparation, semantic search, and vector databases. The small models, such as summarizers, recommenders, and evaluators, are also reliant on CPUs.

Patel and Farhat emphasize that these agents require secure environments with robust isolation for code execution. Google's gVisor, an open-source project, provides this isolation layer.

Google Compute Engine's agents run in a zero-trust environment, necessitating isolation technologies for secure execution. Google's GKE Agent Sandbox can spin up 300 sandboxes per second per cluster, offering scalability for this agentic era. The platform uses pod snapshots and warm pools of suspended environments to help customers scale quickly without over-provisioning.

Google claims Axion can provide up to 2x price performance and 60% better energy efficiency compared to current-generation virtual machines. Customers can choose between C4A machines for high-performance workloads and N4A machines for cost-optimized tasks. Patel emphasizes that customers should consider all options, including CPUs, GPUs, and TPUs, to ensure applications can scale effectively in the fluid compute world of AI agents.

The article concludes that CPUs will continue to play a vital role alongside GPUs and TPUs as AI agents evolve.