Why CPUs still matter in the age of AI agents
When the conversation turns to AI infrastructure, it almost always lands on GPUs and TPUs. The New Stack sat down The post Why CPUs still matter in the age of AI agents appeared first on The New Stack .
In the evolving landscape of AI infrastructure, the focus often remains on GPUs and TPUs, while the CPU, once overlooked, is gaining renewed importance. Bhumik Patel of Arm and Mo Farhat of Google discussed this shift, highlighting that CPUs are becoming more crucial as AI transitions from chatbots to agents. Farhat describes the CPU's role as akin to an air traffic controller, orchestrating the complex operations required for autonomous agents.
Today's advanced language models, with six to eight billion parameters, are outperforming their predecessors, and for specialized tasks, CPUs can manage approximately 25 tokens per second—an ample capacity for agentic workloads. The transition from answering queries to performing actions marks a significant evolution. While large language models predominantly run on accelerators, CPUs shoulder the responsibility of orchestration, data preparation, semantic search, and vector databases.
Arm's Patel emphasizes that the company is concentrating on the software and infrastructure layers necessary to run these workloads efficiently. For agents to execute code securely, they require a sandboxed environment to prevent any potential harm to production systems. Google's solution, gVisor, serves as an isolation layer for these agents, ensuring a zero-trust environment.
Google Compute Engine's GKE Agent Sandbox can provision up to 300 sandboxes per second per cluster, facilitating rapid scaling without the need to fully provision all environments. This efficiency, combined with superior price performance and energy efficiency, positions Axion as a cost-effective solution. Google offers Axion C4A machines for high-performance tasks and N4A machines for balancing performance with cost efficiency.
Patel and Farhat agree that as agents scale, a multi-resource approach will be beneficial. CPUs, GPUs, and TPUs can all coexist and complement each other in the fluid compute world. Customers are encouraged to explore all options, ensuring their applications are built to scale across different types of workloads.
Written by urgent.news from The New Stack's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
