PyTorch 2.15 tackles the 'one bad GPU' problem
Technical debt shoveled aside to make large-scale AI training more resilient
A new version of PyTorch, version 2.15, aims to address the issue of one bad GPU causing the entire training job to start from scratch. This update includes a revised backend for Nvidia's Collective Communications Library (NCCL), which enhances fault diagnosis and recovery when GPUs work together. The original NCCL interface, called ProcessGroupNCCL, had technical debt and could create deadlocks, leading to unpredictable performance.
The updated version of NCCL backend enforces a rule that only one communicator is allowed per process group during initialization, which eliminates crosstalk and simplifies parallelism. If a program requires parallelism, developers must create additional process groups. The library also improves handling of stragglers, allowing for easier reconfiguration when a GPU fails or runs slowly.
A "flight recorder" has been added to help administrators understand what went wrong during a failure. The cleaned-up NCCL backend, also called ProcessGroupNCCL, gives developers a fresh start to add new features. Previous versions of the library will remain as a rollback option in 2.15 but will be removed in the upcoming 2.16 release.
PyTorch, a machine learning framework, is used by major AI labs like OpenAI, DeepMind, Meta, and Nvidia for various AI tasks. The new version of PyTorch includes other features like a Python transport API for asynchronous data movement, improved parallel processing for Apple devices, and support for Nvidia's upcoming Blackwell GPUs.
Written by urgent.news from The Register's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
Also reported by 1 other outlet
- PyTorch 2.15 tackles the 'one bad GPU' problem theregister.com
