From private cloud to private AI cloud, software decides who wins

As enterprises experiment with cloud AI, data-control, sovereignty, and token-cost concerns have pushed intelligence back to infrastructure enterprises own. This has led to a rebuild of the private cloud as an AI platform, with production inference dictating requirements. Chris Wolf, global head of AI and advanced services at Broadcom Inc., explained that IT operations teams must now serve frontier models, small local models, and swarms of agents from the same hardware pool.

Balancing these workloads to avoid rising server, energy, and licensing costs is the central architectural problem.

Wolf emphasized that while frontier models should be used where deep reasoning is needed, specialized models and local SLMs (small language models) should also be employed when appropriate. The industry is witnessing a broad coverage of models, placing the burden on IT operations teams.

Agentic workloads introduce new demands, such as warm pools of isolated virtual machines to spin up agents without escapes or privilege escalations. Memory constraints, including key-value cache placement across GPUs and storage class tiering, add to the challenge. Broadcom has positioned its VMware Cloud Foundation as the pooling layer to address these issues.

However, Wolf cautioned against starting with hardware purchases and leaving software considerations for later. He stressed that software choices are crucial for maintaining autonomy over accelerators and ensuring flexibility between cloud and local models. Broadcom's AI factory approach provisions from bare metal through model runtimes and exports a YAML file for cloning additional clusters. Wolf noted that many customers have experienced buyer's remorse when deploying what they thought was a full turnkey solution.

Sovereignty concerns, driven by governments in North America, Europe, and Asia, are another driver. Enterprises must architect for potential change, focusing on software at the forefront of decision-making.