{
  "id": 4702504,
  "title": "Cutting ASR Inference Cost with NVIDIA MPS on Amazon EC2",
  "url": "https://urgent.news/2026/08/31/cutting-asr-inference-cost-with-nvidia-mps-on-amazon-ec2",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-31T19:38:06.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/noah_taro_1e3297725e3fcbf/cutting-asr-inference-cost-with-nvidia-mps-on-amazon-ec2-13an"
  },
  "original_language": "en",
  "account": "When deploying an ASR pipeline in production, not only is speed important, but also how many requests per second can be handled on a single GPU before latency begins to degrade. This post, a collaboration between AWS, NVIDIA, and a Deep Learning Architect at AWS, details how using NVIDIA MPS on Amazon EC2 can reduce inference costs by up to 75%. The key to GPU sharing for inference is utilizing multiple GPUs efficiently, and there are three common methods: time-slicing, MIG, and NVIDIA CUDA MPS. NVIDIA CUDA MPS is a binary-compatible alternative implementation of the CUDA API which allows existing CUDA-based workloads to benefit from concurrent execution without changing the application's programming model. The inference pipeline in this work runs on Amazon EC2 g6e.4xlarge and g7e.4xlarge instances, both of which have NVIDIA L40S GPUs with 48 GB of memory. The pipeline is built from three containerized components, which makes the system modular and easier to benchmark. To build the deployment image, you need the NVIDIA Triton Inference Server container, which is the foundation of the serving side of the workflow. Once built, the image is tagged and ready for use. The best operating point for the configuration is the last concurrency level where the mean latency stays below 650 ms and the p99 latency stays below 1,000 ms. This is important because increasing concurrency can increase throughput, but it can also result in higher latency, which may make it unsuitable for production ASR scenarios. The key takeaway is that increasing concurrency until utilization improves, while monitoring both mean and p99 latency, is a more practical approach than simply maximizing throughput. This cost reduction is meaningful because it comes from improving GPU sharing rather than changing the model or the pipeline itself. However, it is important to remember that there are limits to how far you can push concurrency before latency becomes a problem. Leftover volumes from the experiment, such as model checkpoints and TensorRT cache, should be cleaned up to avoid unnecessary costs and confusion with future runs.",
  "summary": "When an ASR pipeline is pushed to production, the interesting question is not only how fast it runs, but how much throughput you can extract from each GPU before latency starts to break. In the setup described here, that tradeoff was the main lever for reducing inference cost by 75% using NVIDIA MPS on Amazon EC2. This post is a collaboration between AWS, NVIDIA, and Heidi. It also includes input…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}