Voice Robot for Outbound Calls: A Latency Budget from ASR to TTS

A voice robot used for confirming appointments can appear accurate yet still feel broken to the customer. The caller responds with agreement, only to experience a two-second silence before the robot begins speaking. Despite the transcript appearing correct, the call experience is flawed. Latency plays a significant role in this experience; people are accustomed to brief pauses and begin filling the silence after approximately one second.

Instead of asking which speech model is best, it's crucial to focus on where the milliseconds are spent between the customer's speech and the robot's response. This post breaks down the call path into stages, assigns a budget to each, and identifies the challenges that gradually accumulate this latency. The key metric to measure is the time from the end of the user's speech to the first audio byte played back, termed as turn latency.

This metric encompasses all components, rather than focusing individually on ASR processing time, LLM time, or TTS real-time factor. The primary focus should be on achieving a turn latency of 700 ms or less, feeling natural like a quick human operator, while ensuring that a high percentage of calls remain seamless. The outbound call process can be divided into several stages: SIP trunk, media server (RTP), VAD/endpointing, streaming ASR, dialog logic (rules or LLM), streaming TTS, and RTP back to the caller.

By breaking down the call path and optimizing each stage, the robot can provide a more seamless experience.