This week at Romi #2 - Latency and Voice AI

Welcome back to This at Romi. We've been incredibly busy, which is why this post comes a bit later than our usual Friday update! When discussing Voice AI, a 200-300 millisecond reduction in latency may seem insignificant. However, in actuality, it's the difference between an awkward robotic delay and a smooth, natural conversation. This week, we took a deep dive into our pipeline to eliminate latency and eliminate outdated code that was hindering our performance. Here's what we changed:

Firstly, we transitioned from synchronous to asynchronous tools. Prior to this update, when a user created a task, the pipeline would halt until the database completed the write operation. By making our tools asynchronous, the voice flow remains uninterrupted while items are saved in the background.

Secondly, we streamlined our deduplication process. Initially, we had seven layers of deduplication to prevent duplicate tasks from reaching the database. However, these safeguards began to conflict with each other and slowed down the pipeline. With stronger prompt reinforcement and a swift backend cleanup, we've reduced it to just two layers.

Thirdly, we bid farewell to thinking models. Our agent heavily relies on fast tool calls, and the additional reasoning delay before speaking introduced by thinking models was unnecessary. We've swapped to a model specifically optimized for speed and efficient tool invocation.

Lastly, we introduced a toggle between Natural Flow and Push-to-Talk. This addition complements the backend speedups we've made. Now, users can choose to either think out loud or speak in noisy environments without the model prematurely triggering unwanted tasks. Sometimes, the biggest performance improvements don't come from writing complex new code; they come from eliminating redundant safeguards, running tasks in the background, and selecting the right model for the job.

What's the most frustrating piece of legacy safety code you've had to clean from your stack?