Benchmarking Serverless GPUs: Modal vs RunPod vs Replicate Cold Starts (2026)
Deploying open-source LLMs (like Llama-3) or real-time Whisper transcription in production often forces a difficult architectural trade-off: keep dedicated GPUs running 24/7 (expensive) or rely on serverless scale-to-zero (cold start latency penalty). To evaluate container spin-up overhead, we benchmarked median cold start latencies and per-second execution costs across the major serverless GPU…
In the realm of deploying open-source large language models (LLMs) and real-time Whisper transcription, providers face an architectural dilemma: opt for dedicated GPUs running continuously at a high cost, or leverage serverless scale-to-zero capabilities despite the penalty of cold start latency. To assess the impact of container spin-up times, a benchmark was conducted across the leading serverless GPU platforms. The results reveal the following:
Modal's A100 (40GB) GPU experienced a median cold start latency of 1.8 seconds, translating to an hourly rate of approximately $2.85 per GPU, indicating that cold starts are feasible. RunPod Serverless A100 (80GB) GPUs took 4.2 seconds to cold start, commanding an hourly rate of about $2.59 per GPU. Replicate's A100 (80GB) GPUs faced a cold start latency of 6.5 seconds, with an hourly cost of $4.14 per GPU.
Together AI's H100 cluster utilized pooled inference, achieving an instantaneous response time without cold start overheads. Lambda Labs' A100 (80GB) VM boot time stood at 45 seconds, costing $1.89 per GPU per hour, but did not support cold starts.
Key insights from the benchmark include Modal's innovative filesystem and memory snapshotting, which reduced container initialization to under 2 seconds, making user-facing on-demand LLM calls viable despite cold start times. For asynchronous batch jobs or high-throughput queues, RunPod Serverless emerged as the most cost-effective solution per GPU second.
Managed inference services like Together AI offer instantaneous responses without the need for engineering around cold starts, though they lack support for custom CUDA extensions or proprietary container code. The comprehensive benchmark dataset, hardware configurations, and testing scripts are accessible at ServerlessGPUBench, while raw benchmark metrics are openly shared on GitHub via the repository awesome-serverless-gpu-latency.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
