Netflix Treats GPU Reliability as a Distributed Systems Problem

Netflix has developed a reliability platform for large-scale GPU training and inference, addressing hardware failures in AI workloads. Large AI training jobs can run for weeks on hundreds of GPUs, with all workers operating in sync. A single failed GPU, network issue, or slow machine can halt or significantly slow the entire workload.

To tackle this, Netflix built its GPU compute platform on Kubernetes, enabling it to detect failures, automatically remove unhealthy capacity, and allow long-running jobs to recover from machine loss. The platform's reliability measures have led to over 99.95% GPU-node availability and a job success rate increase from around 98% to over 99.5%.

Contrary to conventional cloud health checks, which only confirm a virtual machine's existence and reachability, Netflix's GPU checks reveal if a GPU can actually run a distributed training workload. They monitor GPU errors, memory, interconnect, and thermal conditions, as well as network adapter states and high-speed Elastic Fabric Adapter interfaces.

These checks, designed to fail closed, can signal a machine's unhealthy status if a diagnostic tool fails to respond in time. Netflix's platform also features progressive response mechanisms to failures, allowing existing workloads to continue while blocking new work during transient problems. Large jobs use checkpointing and elastic workers, enabling training to resume from the last healthy checkpoint upon machine loss.

The system treats the training job as a distributed system, identifying slow ranks that may be holding the entire job back. Netflix is also utilizing NVIDIA's Multi-Instance GPU (MIG) capabilities, allowing smaller inference workloads to share GPUs while preserving memory-level isolation. This increases utilization compared to dedicating an entire GPU to each workload.

Netflix is also working towards Kubernetes Dynamic Resource Allocation, which would provide more expressive resource requests and topology-aware placement, including appropriate positioning of GPUs, network adapters, and CPUs within the machine.