Tens of thousands of Nvidia GPUs sit idle in Meta's Prometheus supercluster, just waiting for a failure
And other fun facts about how Zuckercorp aims to keep a million GPUs running long enough to train a model
Meta's Prometheus supercluster in New Albany, Ohio, is home to over 11 percent of its GPUs that remain idle or dead, waiting for a failure to occur. This massive pool of redundant GPUs is one of the measures Meta has taken to ensure uninterrupted AI training runs. The cluster, powered by Nvidia's Grace Blackwell and Vera Rubin rack systems, is expected to scale up to a gigawatt capacity with about a million accelerators.
However, as with any large-scale compute cluster, more GPUs mean more potential problems. In a 100,000-GPU cluster, failures are inevitable, and Meta has had to build redundancy into its system to ensure continuity. By keeping some GPUs in reserve, Meta can quickly replace a failed GPU, minimizing the impact of an interruption. Additionally, Meta has implemented a data parallelism model, where multiple copies of the model are resident within the cluster, to mitigate the impact of a hardware failure.
When a faulty replica is identified, it can be taken offline, while the remaining cluster continues productive training. Meta's checkpointing scheme, similar to auto-save or a revision history, allows the model state to be saved asynchronously to multi-tier storage, enabling the system to resume training from the last known good checkpoint.
Despite these measures, Meta is still working to minimize interruptions, aiming to reduce the rate to just two per day for every 8,000 GPUs.
Written by urgent.news from The Register's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
