Why waiting longer makes voice AI worse

Voice agents often determine when a user is done speaking by detecting the absence of speech. This method, known as voice activity detection (VAD), flags the audio as speech or not speech, and a timer runs during the non-speech period. When the timer exceeds a set threshold, the turn concludes, and generation begins. This approach is understandable but flawed, worsening in more natural conversations.

People naturally pause while constructing thoughts instead of simply reading them. These pauses can trigger the VAD to end the turn and start responding, leading to being cut off mid-thought. A quick fix seems to increase the threshold before deciding the turn has ended, allowing for longer pauses. However, this adjustment creates a new problem: latency.

Speech recognition systems have a delay, and when this delay becomes noticeable, users perceive the system as slow and broken, as there is no visual cue to indicate the machine is alive. Raising the threshold stops interruptions but removes the feeling of a real interaction. The underlying issue is that VAD relies solely on silence as a decision signal, which is weak.

Humans detect turn ends using syntax, prosody, and semantics – not just timing gaps. To address this, a different architecture is needed: run a lightweight endpointing model on the partial transcript, outputting a probability of the turn being complete, with silence being just one input. This approach treats silence as one piece of information among others, rather than the sole determinant of the decision.

Another crucial aspect is decoupling detection from commitment. The turn-end prediction does not need to be an irreversible decision. Generation can begin on a probable endpoint and be easily canceled if the user resumes. This transforms a hard classification problem into a soft one, allowing for tolerance of occasional errors. Despite these insights, challenges remain, particularly in barge-in handling, cancellation costs, and managing situations where the user talks over the response.

Open questions persist, and the author expresses a desire to explore these issues further in ongoing discussions. For those who have implemented real-time voice systems, the author is curious to hear about their endpointing implementations and how they handle early predictions firing while generation has already started.