From 9 Seconds of Voice AI Latency to 1.5 Seconds: Building an In-House Voice AI System
Nine seconds of silence. That's how long a caller waited after asking our AI assistant a simple question like "What's the TB test process?" long enough that most people would hang up, assuming the call had dropped. We got that down to about 1.5 seconds. This is the story of how a straightforward platform-integration task turned into designing an in-house Voice AI architecture from scratch and the…
Nine seconds of silence was the wait time callers experienced after asking the AI assistant a simple question, often causing them to hang up. The project began as an attempt to integrate VAPI, a Voice AI platform, into the existing system for automating user onboarding. However, the client's request for a more natural voice led to the realization that customizations would be required beyond what VAPI could provide.
The team explored Twilio, Deepgram, and Cartesia as alternatives, ultimately deciding to build their own Voice AI system to maintain control over the orchestration layer. Optimizing for time-to-first-audio instead of total response time was crucial, and they achieved this by implementing conditional RAG, sentence-level streaming to text-to-speech, warm connections, and on-device embeddings.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
