Your agent's p99 is a different animal
Originally published on Loop & Retry — field notes on building LLM agents that survive production. The demo felt instant. The agent answered in about four seconds, every time you ran it on stage. Then you shipped it, and the support queue filled with "it hangs." Nothing was broken. Your average latency really was four seconds. The problem is that nobody experiences the average — they experience…
This article discusses the difference between the average latency and the worst-case latency (p99) experienced by users when running language model (LLM) agents. The author explains that adding more steps to the agent increases the likelihood of slow steps, which significantly impacts the overall latency. The author provides a Python script to demonstrate this effect, showing that even a single slow step can increase the p99 latency by several seconds.
The article emphasizes that budgeting for timeouts, loading spinners, or SLAs should be based on the p99 latency, rather than the mean latency, as the mean does not accurately represent the experience of the majority of users. The author also highlights the importance of considering the impact of retries, which can serialize additional rounds and become the tail themselves.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
