Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod
Amazon SageMaker HyperPod now offers enhanced observability to help you diagnose GPU training issues faster. It brings together diagnostic and profiling capabilities for training workloads on both Amazon EKS and Slurm orchestrators, pinpointing issues across GPU, network, and host layers.
Large-scale model training on GPUs is prohibitively expensive, and a single malfunctioning accelerator can halt an entire distributed job. Diagnosing the root cause of diminished throughput is often a laborious process that involves combing through logs and disparate dashboards to identify the source. Amazon SageMaker HyperPod now offers enhanced observability to simplify the process of pinpointing GPU training issues.
This new capability integrates diagnostic and profiling features for training workloads running on both Amazon Elastic Kubernetes Service (Amazon EKS) and Slurm orchestrators.
Diagnostic summary leverages Amazon CloudWatch’s DetectGroupOutliers API to compare the utilization, memory, temperature, power, and SM clock of GPUs within a training job. GPUs that deviate from their peers are flagged as outliers, alerting teams to potential issues before they impact training performance. Static diagnostic rules also flag common failure patterns such as GPU hardware faults (XID errors, ECC memory errors), EFA throughput drops, thermal throttling, and overheating, allowing teams to address degradation proactively.
By isolating a single bad accelerator, enhanced observability addresses three critical gaps in visibility. It quickly identifies the culprit, whether it stems from XID errors, ECC memory faults, or thermal throttling, freeing teams to focus on corrective actions instead of searching for the problem. Together, these capabilities significantly reduce troubleshooting time and ensure that expensive accelerators remain productive.
Enhanced observability is delivered as an optional add-on for Amazon EKS-orchestrated clusters and as an extension for Slurm-orchestrated clusters. Default metrics, which are collected automatically, monitor cluster health, CPU/disk/OS usage, GPU utilization, and temperature, with updates every 30 seconds. Advanced metrics, which require opt-in, provide deeper insights through a full node exporter suite, comprehensive NVIDIA GPU DCGM and AWS Neuron metrics, and Elastic Fabric Adapter (EFA) network interface statistics.
Teams can choose between Amazon CloudWatch as the default metric store (offering lower pricing and tight integration with SageMaker Insights) or Amazon Managed Service for Prometheus for those already using Prometheus and Grafana.
Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.