To rank the world's biggest supercomputers, we're gonna need a bigger bench
AI has birthed some truly colossal supers, they just aren't on the Top500
Ranking the world's most powerful supercomputers has become increasingly challenging as the rise of generative AI has shifted focus from a single list to a new reality. China's Lineshine supercomputer, with a staggering 2.2 exaFLOPS of double precision compute, now leads the bi-annual Top500 ranking, while other behemoths like SpaceXAI's Colossus, Meta's Promethius, and OpenAI's Stargate pack hundreds of thousands of GPUs, far surpassing anything on the Top500 today.
These companies are not interested in submitting to the Top500 ranking, which has posed a dilemma for its creators.
Erich Strohmaier, co-author of the Top500 list, believes it's time for a rethink. The problem lies in the fact that capacity has pulled ahead of capability in terms of market size, as compute capability was once closely aligned with capacity. However, in recent years, with the advent of AI, capacity has surpassed capability, making it difficult to measure the performance of large clusters.
The top-ranked supercomputers are designed to run many smaller or medium-sized code instantiations, such as training, fine-tuning, reinforcement learning, and inference, rather than one code across the entire cluster.
To better track these massive superclusters, Strohmaier suggests developing a new benchmark or methodology better suited to judge their capability. This benchmark should prevent operators from gaming the system while still providing a measure of their performance. One possible approach is allowing for lower precisions, as seen with the HPL-MxP benchmark, which supports lower-accuracy floating-point data types like FP8 or FP4.
Another avenue is developing a separate AI benchmark that can keep up with the rapid pace of change in model architecture. However, building such a benchmark is challenging, as the AI workloads change every two weeks, making it difficult to create a benchmark that remains relevant for multiple decades.
Written by urgent.news from The Register's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.