Reduce ASR inference costs by 75% with NVIDIA MPS on Amazon EC2
Serving automatic speech recognition (ASR) models at scale is costly when each request uses only a fraction of a GPU. Learn how NVIDIA CUDA Multi-Process Service (MPS) with NVIDIA Triton Inference Server on Amazon EC2 GPU instances cuts GPU infrastructure by 75% while holding sub-second latency at 92.1 requests per second per GPU.
This article explores how NVIDIA Multi-Process Service (MPS) can reduce automatic speech recognition (ASR) inference costs by 75% when deployed on Amazon Elastic Compute Cloud (Amazon EC2) GPU instances. ASR inference requests typically use only 15-20% of a GPU's compute capacity, leaving 80% idle. This inefficiency requires running 16 GPU instances to sustain sub-second transcription latency at peak traffic.
By leveraging NVIDIA MPS combined with NVIDIA Triton Inference Server on EC2 GPU instances, GPU infrastructure requirements can be reduced by 75% (from 16 instances to 4), while maintaining sub-second latency at 92.1 requests per second per GPU. The article also highlights model-level optimizations with ONNX and TensorRT, as well as request scheduling with Triton, to further improve performance.
Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
