SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

Amazon SageMaker Inference now supports Multi-Instance GPU (MIG) to maximize GPU utilization for model endpoints. Previously, deploying models to GPU endpoints required using an entire accelerator, which could lead to either underutilization or wasted capability. With MIG, NVIDIA splits a physical GPU into independent partitions, each receiving a dedicated share of compute units and memory.

This allows deploying multiple models on a single GPU, while isolating them from one another. The first benefit is being able to deploy more models than you have GPUs, with isolation between them. The second benefit is helping customers choose the right-sized unit for their needs, leading to cost savings. GPU partitioning works on endpoints using inference components, which allow deploying several models to one endpoint and setting the compute each model gets.

Partitioning builds directly on this concept, with each partition being a resource requested by the component, along with CPU and memory. Customers previously had three options when they had more models than accelerators: run fewer, larger endpoints and batch harder, use multi-model endpoints, or move to self-managed Kubernetes.

However, each of these options falls short in some way. MIG eliminates the need for self-managed Kubernetes, as SageMaker AI handles GPU configuration, placement, and scaling. This provides hardware-isolated GPU sharing on a managed endpoint, without the need to run the underlying platform. Salesforce is among the customers who have been instrumental in developing this capability, recognizing the value of GPU partitioning for their inference workloads.