Amazon cracks down on 'CPU waste' among engineers as agentic AI crunch intensifies — CPU demand makes low-utilization EC2 instances a hot commodity

Amazon Web Services (AWS) is taking action against the excessive utilization of EC2 instances by its engineers, amid an intensifying demand for CPU capacity. As per reports from The Information, the tech giant urged engineers in May to reduce CPU waste to ensure sufficient CPU capacity to meet customer demand.

This directive stems from a surge in CPU demand in data centers, with the traditional eight-to-one or four-to-one ratio of GPUs to CPUs nearing parity. The rise in CPU demand is largely attributed to the advent of agentic AI, a new approach to productivity that many companies, including Amazon, find challenging to manage.

Traditional EC2 instances have been used for development purposes, leveraging the relatively low CPU utilization required for web infrastructure. However, engineers are now faced with waiting days for access to an instance, a stark contrast to the hours it used to take. This situation has never been encountered by engineers who have worked at Amazon for several years.

AWS operates various types of CPUs in EC2 instances, including AMD, Intel, and its recently launched Graviton5 chip. Graviton5, Amazon's most powerful CPU to date, adopts an Arm-based architecture, similar to Nvidia's Vera CPU and Arm's own AGI.

The surge in demand for CPUs is mainly due to AI agents, a novel paradigm in productivity that is proving difficult for even large corporations to handle. An instance of a coding agent recently depleted $1.8 million in token costs at Amazon, far surpassing its development budget by 860%.

Agentic workloads, which are more complex than traditional inference workloads, often necessitate tool calls running on the CPU and intricate orchestration of inference on GPUs. This shift has put CPUs at the forefront of the agentic AI era.

Major players in the industry, like Intel, AMD, and Nvidia, are capitalizing on this CPU demand. AMD has recently unveiled its Zen 6 'Venice' CPU line for data centers, marking the first time the company has introduced a new architecture in the data center before the client market in over three decades. Nvidia has also shifted its focus from accelerators to its new Vera CPU to secure its position in the expanding agentic AI infrastructure market.

While Amazon grapples with CPU capacity both internally and among external customers, shortages are primarily affecting spot instances. A consultant cited in The Information report noted that contracted capacity has not experienced any shortages.