{
  "id": 7848046,
  "title": "Fault tolerant distributed training on Amazon EKS using NVRx",
  "url": "https://urgent.news/2026/09/16/fault-tolerant-distributed-training-on-amazon-eks-using-nvrx",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-16T18:59:25.000Z",
  "source": {
    "name": "AWS Machine Learning",
    "slug": "aws-machine-learning",
    "url": "https://aws.amazon.com/blogs/machine-learning/fault-tolerant-distributed-training-on-amazon-eks-using-nvrx/"
  },
  "original_language": "en",
  "account": "Large-scale distributed training jobs often face interruptions due to various faults, such as network issues, memory errors, and software exceptions. When this happens, multiple GPUs can be affected, leading to a cascade of problems including GPU timeouts, pod crashes, and unexpected restarts. This results in wasted expensive GPU hours without any progress in the training process. Traditional synchronous checkpointing adds another layer of inefficiency, blocking all ranks on I/O during saves, consuming up to 40% of total wall time on the cluster.\n\nTo address these issues, this article demonstrates how to integrate NVIDIA Resiliency Extension (NVRx) into PyTorch Fully Sharded Data Parallel (FSDP) training on Amazon Elastic Kubernetes Service (Amazon EKS). The authors outline three key features of NVRx: async checkpointing, in-process restart, and in-job restart using ft_launcher. These features aim to solve the problems of interruption and fault tolerance in distributed training on Amazon EKS.\n\nThe solution combines NVRx fault tolerance primitives with an EKS-based training environment tailored for high-performance multi-node GPU workloads. NVRx handles application-level resilience, while the EKS cluster provides the underlying infrastructure such as GPU scheduling, high-bandwidth networking, and shared storage for checkpoint persistence. NVRx consists of three features: async checkpointing, in-process restart, and ft_launcher. Async checkpointing allows the state dictionary to be handed to a background process, preventing bottlenecking at rank-0 and enabling independent shard writes. In-process restart allows transient faults like unhandled exceptions or NCCL hangs to be caught without killing the Python process. ft_launcher is a binary that handles hard faults like SIGKILL or out-of-memory (OOM) kills by respawning fresh workers in the same job.\n\nThe authors tested these features on an EKS cluster with H100 GPUs, ranging from 2 to 8 nodes. They used Amazon FSx for Lustre as the shared filesystem for checkpoint I/O, ensuring minimal read latency during recovery. By employing these techniques, the authors aim to provide a robust and efficient solution for distributed training on Amazon EKS, minimizing interruptions and maximizing training progress.",
  "summary": "Integrate NVIDIA Resiliency Extension (NVRx) into PyTorch FSDP training on Amazon EKS to overlap checkpoint I/O with training and recover from GPU faults in seconds. This post covers async checkpointing, in-process restart, and ft_launcher in-job restart, with H100 benchmarks at 2 to 8 nodes showing 99%+ training efficiency and second-scale recovery.",
  "key_points": [
    "NVRx integrates into PyTorch FSDP training on Amazon EKS to improve fault tolerance.",
    "Async checkpointing, in-process restart, and ftlauncher are NVRx's key features.",
    "Tested on EKS cluster with H100 GPUs, achieving robust and efficient distributed training."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}