High-severity Nvidia bug could crash GPU monitoring on exposed servers

A recent security flaw affecting Nvidia's GPU servers has the potential to crash GPU monitoring services and disrupt AI workloads. Researchers at Lava, a datacenter security startup, discovered that thousands of GPU servers were exposing the Nvidia DCGM Exporter to the internet, with hundreds possibly vulnerable to this high-severity flaw.

This flaw allows unauthenticated attackers to crash the GPU monitoring service, providing detailed information about GPU infrastructure for reconnaissance purposes. The DCGM Exporter collects telemetry data from GPUs, including hardware, utilization, memory usage, power consumption, and error events. Each GPU has a unique identifier, or UUID, and all these metrics are exposed in plaintext over HTTP, providing attackers with valuable information.

Researchers found approximately 2,100 exposed GPU servers, which included 12,000 GPU UUIDs from about 300 organizations. Nearly half of these exposed GPUs were located in the US, representing around $100 million worth of hardware, including Nvidia Blackwell Ultra B300, H200, and H100 GPUs, as well as consumer RTX 5090 and 4090 systems.

The security flaw could lead to memory exhaustion and crashes in the monitoring service, impacting AI training or inference workloads. Nvidia released a fix for the flaw in version 4.8.2, and operators are advised to upgrade to the latest version or later. Additionally, researchers also discovered that about 25% of exposed DCGM hosts also revealed data from Go’s /debug/pprof/ built-in profiling tool, which collects and exposes runtime performance data.

This further increases the risk of disruption to AI workloads. Lava reported the findings to the affected providers, including Nebius, Voltage Park, Lambda, Northern Data, and DigitalOcean, and they worked with customers to address the exposures.