GPUs on Kubernetes are bring-your-own-everything, starting with the bill

Running Kubernetes platforms often involves monitoring workloads, which can be done through Grafana dashboards. These dashboards show what containers requested versus what they used, allowing for easy identification of resource consumption by individual deployments. This method has been effective for CPU and memory resources for years. However, when it comes to GPUs, the visibility and management become more complex.

GPU resource visibility requires creating custom metrics, as GPUs don't provide the same level of granular visibility as CPU and memory. NVIDIA's DCGM exporter, a popular tool for GPU metrics, reports at the physical device level by default. This approach works well when a single pod has exclusive access to a GPU, but it becomes problematic when multiple pods share a GPU through time-slicing or Multi-Instance GPU (MIG).

With time-slicing or MPS, DCGM provides the same value for all pods sharing a GPU, making it difficult to attribute resource usage accurately. This issue is further exacerbated by the fact that GPUs are billed as whole numbers, requiring precise matching of requests and limits without any overcommitting at the scheduler level. As a result, teams must build their own rightsizing math to determine how much of a GPU's capacity a workload actually used before addressing inefficiencies.

Another challenge with GPU resource management lies in determining GPU utilization. While CPU utilization is straightforward, GPU utilization is more nuanced. The traditional utilization metric indicates whether a kernel was running on the GPU during a sample window, but it doesn't provide information on actual compute or memory bandwidth consumption. GPUs can sit at 100% utilization while barely performing any meaningful work, and memory constraints often become the first signal of resource exhaustion.

To address these challenges, teams must develop their own definitions of utilization, focusing on metrics such as compute occupancy, memory footprint, bandwidth, or power consumption. By relying on these signals, organizations can make more informed decisions about resource allocation and billing. NVIDIA's NVML library can provide per-process memory usage on a GPU, but the data may not always add up to the total GPU capacity due to the driver reserving memory for its own use.

In shared GPU environments, accurate attribution of resource usage becomes even more challenging, requiring careful consideration of billing strategies and the adoption of tools that can provide more accurate GPU utilization metrics.