Urgent.News

What's breaking now, across thousands of outlets.

AI

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

Large training jobs can stall or compute wrong results even when every node looks healthy. Amazon SageMaker HyperPod now adds workload recovery and expanded infrastructure checks that catch these failures sooner and raise the goodput of your training workloads.

Amazon SageMaker HyperPod is introducing workload recovery and expanded infrastructure checks to improve the reliability and efficiency of training workloads. These new features aim to catch issues that can lower goodput, the portion of a cluster's time dedicated to productive training. Existing HyperPod resiliency measures focus on individual nodes, checking for GPU errors, deep health checks, and automatic node recovery.

However, these measures cannot detect if GPUs compute correctly, if a job is making progress, or if the Kubernetes components a job depends on are functioning properly.

Three new capabilities address these limitations:

1. MultiNodeTrainingValidation: This deep health check trains a small model on instances as they're provisioned or on demand. If the loss becomes NaN or infinite, or if training doesn't finish in time, the instance is considered faulty. The validation is available on supported EFA-enabled NVIDIA GPU instances within EKS clusters.

2. Hung job detection: This agent monitors Slurm jobs, checking for stopped progress or failed training jobs. Upon detecting such an issue, it sends an event to CloudWatch and can cancel the job, allowing Slurm to requeue it. Hung job detection is now available for Slurm clusters.

3. HyperPod Kubernetes Resource Monitoring Agent: This agent continuously monitors the Kubernetes layer of an EKS cluster, looking for issues such as crash-looping device plugins, missing DaemonSets, and tainted nodes. When a job fails, it reports the involved pods and nodes as Kubernetes events and logs.

These new capabilities improve scheduling goodput (availability of resources needed), runtime goodput (progress made), and program goodput (hardware peak performance utilization) by proactively identifying and addressing problems before they impact training quality and efficiency.

Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at aws.amazon.com →

More in AI

More from Wednesday 18 November →