Fault tolerant distributed training on Amazon EKS using NVRx
Integrate NVIDIA Resiliency Extension (NVRx) into PyTorch FSDP training on Amazon EKS to overlap checkpoint I/O with training and recover from GPU faults in seconds. This post covers async checkpointing, in-process restart, and ft_launcher in-job restart, with H100 benchmarks at 2 to 8 nodes showing 99%+ training efficiency and second-scale recovery.
Large-scale distributed training jobs often face interruptions due to various faults, such as network issues, memory errors, and software exceptions. When this happens, multiple GPUs can be affected, leading to a cascade of problems including GPU timeouts, pod crashes, and unexpected restarts. This results in wasted expensive GPU hours without any progress in the training process.
Traditional synchronous checkpointing adds another layer of inefficiency, blocking all ranks on I/O during saves, consuming up to 40% of total wall time on the cluster.
To address these issues, this article demonstrates how to integrate NVIDIA Resiliency Extension (NVRx) into PyTorch Fully Sharded Data Parallel (FSDP) training on Amazon Elastic Kubernetes Service (Amazon EKS). The authors outline three key features of NVRx: async checkpointing, in-process restart, and in-job restart using ft_launcher. These features aim to solve the problems of interruption and fault tolerance in distributed training on Amazon EKS.
The solution combines NVRx fault tolerance primitives with an EKS-based training environment tailored for high-performance multi-node GPU workloads. NVRx handles application-level resilience, while the EKS cluster provides the underlying infrastructure such as GPU scheduling, high-bandwidth networking, and shared storage for checkpoint persistence.
NVRx consists of three features: async checkpointing, in-process restart, and ft_launcher. Async checkpointing allows the state dictionary to be handed to a background process, preventing bottlenecking at rank-0 and enabling independent shard writes. In-process restart allows transient faults like unhandled exceptions or NCCL hangs to be caught without killing the Python process. ft_launcher is a binary that handles hard faults like SIGKILL or out-of-memory (OOM) kills by respawning fresh workers in the same job.
The authors tested these features on an EKS cluster with H100 GPUs, ranging from 2 to 8 nodes. They used Amazon FSx for Lustre as the shared filesystem for checkpoint I/O, ensuring minimal read latency during recovery. By employing these techniques, the authors aim to provide a robust and efficient solution for distributed training on Amazon EKS, minimizing interruptions and maximizing training progress.
Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
