Huawei unveils 'SuperCluster' architecture designed to make 1 million AI chips work as a single computer

Huawei has unveiled a new computer architecture called Peerium, designed to allow up to a million AI chips to function as a single computer. This innovation surpasses the capabilities of existing AI training systems, such as SpaceXAI's Colossus, which utilizes 200,000 interconnected chips. Traditional AI training typically involves separate servers for chips, with data communication over networks.

However, Peerium employs a high-speed interconnect system to maintain efficient communication across multiple racks, even as the number of chips climbs into the hundreds of thousands.

Huawei unveiled Peerium at its annual CONNECT conference on September 17. The architecture's performance hinges on three key elements: nested parallelism, unified memory addressing, and peer-to-peer interconnects. Nested parallelism breaks down workloads into smaller tasks organized hierarchically across groups of chips, similar to organizational structures coordinating tasks.

Unified memory addressing enables processors to access memory across other servers, unlike traditional setups where processors only access their attached servers' memory. Peer-to-peer interconnects allow direct communication between components, eliminating the need for intermediaries.

These principles collectively aim to optimize resource utilization, enabling a single system to coordinate larger numbers of chips to tackle demanding AI workloads. Huawei's interconnect technology, UnifiedBus, boasts an inter-rack communications bandwidth of up to 800GB per second, surpassing existing technologies like NVIDIA's NVLink and InfiniBand, which offer 200GB per second. With one million processors working as one computer, Huawei envisions improved and more efficient AI training and inference capabilities.

Huawei also introduced the Atlas 950 SuperPoD, a computing system built on Peerium and powered by over 8,000 Huawei Ascend 950DT AI accelerator chips. The company expects to start supplying SuperPoDs by early next year and plans to deploy SuperClusters, linking multiple SuperPoDs, later.