{
  "id": 18546725,
  "title": "PyTorch 2.15 tackles the 'one bad GPU' problem",
  "url": "https://urgent.news/2026/11/06/pytorch-2-15-tackles-the-one-bad-gpu-problem",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-06T18:36:42.000Z",
  "source": {
    "name": "The Register",
    "slug": "the-register",
    "url": "https://www.theregister.com/devops/2026/11/06/pytorch-215-tackles-the-one-bad-gpu-problem/5308227"
  },
  "original_language": "en",
  "account": "A new version of PyTorch, version 2.15, aims to address the issue of one bad GPU causing the entire training job to start from scratch. This update includes a revised backend for Nvidia's Collective Communications Library (NCCL), which enhances fault diagnosis and recovery when GPUs work together. The original NCCL interface, called ProcessGroupNCCL, had technical debt and could create deadlocks, leading to unpredictable performance. The updated version of NCCL backend enforces a rule that only one communicator is allowed per process group during initialization, which eliminates crosstalk and simplifies parallelism. If a program requires parallelism, developers must create additional process groups. The library also improves handling of stragglers, allowing for easier reconfiguration when a GPU fails or runs slowly. A \"flight recorder\" has been added to help administrators understand what went wrong during a failure. The cleaned-up NCCL backend, also called ProcessGroupNCCL, gives developers a fresh start to add new features. Previous versions of the library will remain as a rollback option in 2.15 but will be removed in the upcoming 2.16 release. PyTorch, a machine learning framework, is used by major AI labs like OpenAI, DeepMind, Meta, and Nvidia for various AI tasks. The new version of PyTorch includes other features like a Python transport API for asynchronous data movement, improved parallel processing for Apple devices, and support for Nvidia's upcoming Blackwell GPUs.",
  "summary": "Technical debt shoveled aside to make large-scale AI training more resilient",
  "key_points": [
    "PyTorch 2.15 addresses issue of one bad GPU causing training job to restart",
    "Revised NCCL backend enhances fault diagnosis and recovery in GPU parallelism",
    "Flight recorder added for easier understanding of GPU failure during training"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}