How Deepgram optimized voice AI at scale with AWS Trainium
Deepgram worked with the AWS Forward Deployed Engineering team to port its Nova-3 speech-to-text model to AWS Trainium2 for offline batch inference. This post covers porting to the AWS Neuron stack, kernel-level tuning, a data-parallel Rust runtime, and the throughput and cost results on batch transcription.
Deepgram, a company specializing in voice AI, has optimized its speech-to-text models for scalable inference using Amazon Web Services (AWS) Trainium2 chips. This optimization has enabled Deepgram to process quadrillions of audio tokens per year for over 200,000 developers and 1,400 organizations. The company serves audio through APIs or on Amazon SageMaker AI.
The key to Deepgram's success lies in the efficient utilization of accelerator hours. To achieve this, they worked with AWS to bring their Nova-3 speech-to-text model onto Trainium2 for offline batch workloads like contact-center analytics and medical transcription. The team started by delivering Nova-3 as standard PyTorch code and then ported it to Trainium2 in PyTorch reconstruction, ultimately tuning it for the new hardware.
The optimized configuration runs eight model replicas on a single Amazon EC2 trn2.3xlarge instance, processing 11–70% more audio per dollar compared to their previous compute options. These improvements help Deepgram scale efficiently, maintain better model shipments, and offer lower prices to customers. The benefits extend to customers who self-host Deepgram models in their own virtual private cloud (VPC) on Amazon SageMaker AI, where the company continues to improve ease of setup, reliability, and observability.
The optimization process began with standard PyTorch code, where TorchNeuron's native PyTorch integration of the Neuron Compiler was used. The model was first run in eager mode on Trainium2, with operations falling back to the CPU identified and then compiled with torch.compile. The encoder and decoder were compiled as separate static-shape graphs using torch.compile.
Throughput-maximizing batch sizes were determined by sweeping batches 1–128, and several layers were replaced with fused kernels from the public NKI Library, targeting Deepgram's offline inference workloads. This optimization process involved identifying bottlenecks, replacing kernels, profiling workloads, and tuning kernels for better performance. The final configuration achieved up to 50x speedup on the convolution module at batch sizes ≤32.
Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
