Microsoft bets on memory that moves to the workload as AI supply tightens

Memory is in high demand in AI data centers, prompting hyperscalers to explore new methods for efficiently distributing it. Composable memory, which unites capacity beyond individual servers and assigns it to workloads on demand, is seen as a promising solution to this challenge. At Microsoft, the company's team is investigating system-level technologies that can transform memory into a shared, dynamically allocated resource across Azure, according to Samir Rajadnya, a principal architect at Microsoft.

Rajadnya explained that the idea behind composable memory is to treat it as an essential resource rather than just a servant to CPUs or GPUs. As AI demand outpaces production, memory scarcity is becoming increasingly visible in AI factories where GPUs are idle due to a lack of memory and storage. Rajadnya highlighted that Azure, functioning as a utility for companies like Delta Air Lines Inc. and Walmart Inc., requires efficient memory utilization while maintaining reliability and security.

To address this, Rajadnya suggested focusing on the top two or three critical workloads and designing systems tailored to them, emphasizing the importance of vertical integration among workload, system, software, and silicon. The interconnects between memory and processors, including specialized accelerators, play a crucial role in enabling composable memory.

With optical connections becoming increasingly important, Microsoft is showcasing its AI infrastructure work at the Open Compute Project Global Summit to foster innovation in this area.