I ran a GPU inference app for a month on Azure serverless GPU. Here's the actual bill.

I tested a GPU inference application on Azure serverless GPU for a month. The service, running on an NVIDIA T4, was billed based on actual usage. The total cost came to around $13/month, even when the app was idle for most of the time. However, the most significant expense was not the GPU itself, but the storage costs. The app ran on a min-replicas=0 setup, scaling to zero replicas when idle.

The cold start time for the GPU was approximately 5 minutes, which is an important factor to consider for an internal tool where users expect a brief delay on first use. For a customer-facing endpoint, the cold start may not be acceptable. The biggest line item on the bill was the Azure Container Registry, costing about $11/month.

This was due to the large CUDA base images and model weights, which contributed to high storage costs. To reduce costs, the author implemented housekeeping measures, such as keeping only one deployed image, removing old tags, and performing regular cleanup. Despite the initial challenges, such as setting up the deployment and addressing issues with the Azure CLI, the author found that serverless GPU is a cost-effective solution for bursty, low-frequency workloads.