I ran a GPU inference app for a month on Azure serverless GPU. Here's the actual bill.
I run a small internal speech-synthesis service on an NVIDIA T4 via Azure Container Apps' serverless GPU, with min-replicas=0 . The pitch is that you pay nothing while nobody's using it. I wanted to know if that's actually true, so I pulled the real numbers out of Azure Cost Management, scoped to just that resource group. Short answer: yes, it's true. But the biggest line item on my bill wasn't…
I tested a GPU inference application on Azure serverless GPU for a month. The service, running on an NVIDIA T4, was billed based on actual usage. The total cost came to around $13/month, even when the app was idle for most of the time. However, the most significant expense was not the GPU itself, but the storage costs. The app ran on a min-replicas=0 setup, scaling to zero replicas when idle.
The cold start time for the GPU was approximately 5 minutes, which is an important factor to consider for an internal tool where users expect a brief delay on first use. For a customer-facing endpoint, the cold start may not be acceptable. The biggest line item on the bill was the Azure Container Registry, costing about $11/month.
This was due to the large CUDA base images and model weights, which contributed to high storage costs. To reduce costs, the author implemented housekeeping measures, such as keeping only one deployed image, removing old tags, and performing regular cleanup. Despite the initial challenges, such as setting up the deployment and addressing issues with the Azure CLI, the author found that serverless GPU is a cost-effective solution for bursty, low-frequency workloads.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
