How we made a text-to-speech model respond in sub-50 ms

The text-to-speech (TTS) model Qwen3-TTS 1.7B CustomVoice implementation demonstrated impressive performance, achieving 10 requests per second (RPS) and sub-50 milliseconds (ms) p95 time-to-first-audio (TTFA) on a single NVIDIA H100 SXM SXM. To evaluate various TTS implementations, five models were compared under open-loop Poisson traffic: the proposed system, vLLM-Omni, SGLang-Omni, VoxServe, and M*.

After tuning each model for low-latency streaming, the proposed system was the only one to achieve sub-50 ms p95 TTFA at 10 RPS, with performance remaining below 100 ms even at 20 RPS. At a cost of $4.29 per hour for the H100 SXM instance, the system translates to roughly $2 per 1 million characters at full utilization. In comparison, ElevenLabs V3 costs $100 per 1 million characters, while Cartesia Sonic 3.5 costs $49 per 1 million characters at higher TTFA.

The system's success is attributed to a four-part approach addressing low p95 audible TTFA, zero underruns, high RPS, and real-time playback on a single NVIDIA H100 SXM. The Qwen3-TTS model is chosen due to its popularity and permissive license. The system focuses on low TTFA, continuity, quality, and capacity while tuning each serving engine separately.

A key contribution is the introduction of a dynamic trim that detects sustained speech, removes silence before onset, and streams the remaining audio, improving TTFA by approximately 80ms. The system also optimizes codec frame collection and chunk sizes for improved TTFA while maintaining a balance between responsiveness and playback safety.

The proposed system outperforms competitors like VoxServe, which reaches sub-50 ms TTFA at 1 RPS, while other engines only achieve this at higher RPS rates. The architecture of Qwen3-TTS involves a three-part model performing hierarchical multi-codebook generation: Talker, Code Predictor, and Codec. The proposed system takes this further by exposing these modules as independently schedulable tasks, managed by a single scheduler.

This design allows for better coordination of heterogeneous tasks, enabling shorter units of work and more opportunities for the scheduler to interleave requests based on urgency. The scheduler prioritizes requests that have not produced their first audio chunk while establishing streaming for established requests only as they approach playback deadlines. This approach helps maintain batching efficiency while meeting critical request deadlines.