{
  "id": 15806424,
  "title": "Tens of thousands of Nvidia GPUs sit idle in Meta's Prometheus supercluster, just waiting for a failure",
  "url": "https://urgent.news/2026/10/20/tens-of-thousands-of-nvidia-gpus-sit-idle-in-metas-prometheus",
  "topic": "science",
  "section": "Science",
  "published": "2026-10-20T20:39:42.000Z",
  "source": {
    "name": "The Register Science",
    "slug": "the-register-science",
    "url": "https://www.theregister.com/systems/2026/10/20/tens-of-thousands-of-nvidia-gpus-sit-idle-in-metas-prometheus-supercluster-just-waiting-for-a-failure/5304269"
  },
  "original_language": "en",
  "account": "Meta's Prometheus supercluster, located in New Albany, Ohio, houses over 11 percent of its Nvidia GPUs that are idle or waiting for a failure to occur, according to a recent revelation. This massive pool of redundant GPUs is one of several measures Meta has implemented to ensure its AI training runs remain uninterrupted. Prometheus is among Meta's largest compute deployments, with plans to scale to a gigawatt of capacity spanning about a million accelerators. However, building such a large cluster inevitably leads to more problems, as each additional GPU increases the likelihood of failures. Despite this, Meta has taken steps to minimize the impact of these interruptions. During the first phase of Prometheus deployment, the company experienced roughly 1,000 interruptions per day for every 8,000 GPUs. Now, Meta claims to have reduced this number by 99 percent, to about 11 interruptions per day per 8,000 GPUs. To prevent a single failed GPU from halting the entire training run, Meta has set up a system of hot spares, with eight out of every 72 GPUs in each NVLink domain kept as backups. When a GPU fails, one of these spares can immediately take its place, minimizing the disruption. Meta also uses data parallelism in its training jobs, distributing the workload across multiple copies of the model within the cluster. If a fault can be isolated to a data-parallel model replica, the system can take that faulty replica offline while the remaining cluster continues training. This approach converts potential system halts into minor, temporary reductions in throughput. Meta's checkpointing architecture enables it to save the model state, including gradients and optimizer data, to multi-tier storage, allowing for a swift restart in case of a failure. The company has also implemented a comprehensive telemetry system to address NVLink fabric challenges, such as signal degradation and power consumption spikes, in order to maintain more than 85 percent goodput for pretraining and more than 90 percent for reinforcement learning on its latest model, Muse Wonder.",
  "summary": "And other fun facts about how Zuckercorp aims to keep a million GPUs running long enough to train a model",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 2,
    "also_reported_by": [
      {
        "outlet": "The Register",
        "title": "Tens of thousands of Nvidia GPUs sit idle in Meta's Prometheus supercluster, just waiting for a failure",
        "url": "https://urgent.news/2026/10/20/tens-of-thousands-of-nvidia-gpus-sit-idle-in-metas-prometheus-15808075",
        "published": "2026-10-20T20:39:42.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}