Data centers use as little as 60% of power consumed for actual AI work, Nvidia claims — failures and restarts 'consume' a whopping 10% of every watt used

Nvidia asserts that only 70% of power consumed by an AI data center is utilized for actual AI work, with the remaining 30% attributed to inefficiencies. These inefficiencies stem from various sources including cooling, power distribution, failures, and restarts. While a chip shortage was initially considered the primary bottleneck for AI data center capacity, it is now recognized that a data center's connection to the grid plays a significant role.

The inefficiency of AI data centers has been highlighted by data center builder Vertiv, which is collaborating with Nvidia on co-designing the next generation of sustainable facilities. According to Nvidia's Vice President of Supercomputing, AI Cloud Infrastructure, and AI/HPC for EMEA, Rod Evans, the 30% inefficiency comprises 20% from cooling, power distribution, and conversion, and an additional 10% from incorrectly configured racks, failures, and restarts.

Nvidia believes that up to 40% of the 30% inefficiency can be recovered by optimizing the utilization of existing capacity. The primary bottleneck affecting AI data centers is not a shortage of chips but a lack of power. This can be exacerbated by geographical factors, as demonstrated by Vertiv's EMEA president, Paul Ryan, who notes that some locations have power available but no permits, while others have permits granted quickly but lack power. Additionally, remote locations may pose challenges in delivering the necessary equipment.