Cut GPU inference cold start from 8 minutes to less than a minute

We have achieved a dramatic reduction in GPU inference cold start time, cutting it from an average of eight minutes to less than a minute. This significant improvement was achieved through a comprehensive analysis of the six sequential phases that contribute to the model startup time. By identifying the bottlenecks in each phase and implementing targeted fixes, we were able to achieve substantial reductions in cold start time.

For smaller models (up to 100 GB), the compilation phase of the model dominates the startup time. However, for larger models (over 100 GB), network transfer of weights becomes the primary contributor. This means that any optimization targeting a single layer of the cold-start stack has a limited impact, as the overall time is dominated by a different layer depending on the size of the model.

The six phases that contribute to cold start time are as follows:

1. Node provisioning

2. GPU driver initialization

3. Container image pull

4. Model weights download

5. GPU kernel compilation

6. Engine initialization

Each of these phases has its own unique bottleneck and requires a distinct set of fixes to improve cold start performance. The good news is that the improvements we achieved are applicable to any Kubernetes cluster, as they rely on environment variables and volume mounts that can be implemented universally.

For smaller models, the optimization primarily focuses on reducing the time spent on model compilation. For larger models, the emphasis is on minimizing the time required for downloading and loading the model weights from object storage. By addressing these specific bottlenecks, we were able to achieve significant reductions in cold start time, with reductions ranging from 80-93% for models running on existing nodes, and 40-65% for newly provisioned nodes.

This improvement demonstrates the significant progress that has been made in the Kubernetes ecosystem for the inference stack. The adoption of Stable OCI image volumes, Dynamic Resource Allocation (DRA), and Gateway API inference-aware routing extensions have all contributed to this advancement. However, it is important to note that these technologies do not address the entire cold-start stack, as there are still multiple layers that require targeted optimizations for maximum performance gains.