How We Doubled Our Cache Hit Rate When Pre-Warming Wasn’t Enough
Quick question. If you run a voice agent, do you know your prompt cache hit rate on turn 1 ? Not the average across the call. Just turn 1, when the caller has said hello and is sitting in silence waiting for your agent to speak. We thought ours was fine. We pre-warm before every call, like everyone tells you to. Then we measured it properly, and our outbound calls were doing less than half as…
The article discusses the challenges faced by a company running voice agents, specifically how their cache hit rate on turn 1 was lower than expected, despite pre-warming before every call. The key insight is that the cache hit rate was not optimized across calls but only within a single call, due to three rules that had to be met for caching to be effective.
These rules include exact prefix matching, the prefix being long enough, and landing on the same machine for caching to be reused. The author found that their caching optimization was focused on one call at a time, rather than across calls, which led to a lower overall cache hit rate of around 40%.
Brief written by urgent.news from Dev.to's own syndicated text. Machine-written — may contain errors; check the original before relying on it.
