Enterprise IT infrastructure isn’t built for AI inference

Admittedly, the current state of enterprise IT infrastructure is not designed for AI inference. While global AI spending has reached over $300 billion, much of that investment has been channeled toward training models. However, inference workloads are now the dominant focus, with Deloitte reporting that inference accounts for roughly two-thirds of compute, and McKinsey projecting it will represent between 30% and 40% of data center demand by 2030.

Training and inference have fundamentally different requirements. Training thrives on centralized data GPU clusters for maximum power density and interconnects, while inference demands geographically distributed compute capable of responding to real-time user demands. This shift calls for a new approach that prioritizes geographically distributed edge sites, low-latency network connections, and commodity ethernet.

The cost structure of inference is also markedly different from that of training. While training is a one-time capital event, inference is an ongoing operational cost. As autonomous AI agents create new strain on the architecture, the demand for supporting infrastructure will continue to grow. Gartner predicts that by the end of 2026, 40% of enterprise applications will be integrated with task-specific agents, and token and API call volumes will multiply by thousands by 2027.

Moreover, inference is exacerbating the traffic multiplier issue in enterprise WANs and data center networks, which are currently ill-equipped to handle burst traffic patterns. Organizations that fail to build a dedicated infrastructure from the ground up risk unsustainable ROI, with only 28% of AI use cases meeting their ROI projections, according to Gartner's most recent survey.