What our averages were hiding from us

Our latency dashboard reported an average response time of 120ms, a figure that management found satisfactory. However, support teams were inundated with complaints from customers who found the app excruciatingly slow. The discrepancy between the two perspectives highlighted the deceptive nature of averages. An average is a singular figure that seeks to represent an entire group, influenced heavily by the majority.

In this case, 95% of requests were swift, while 5% took a staggering eight seconds. Despite this, the average remained largely unchanged, concealing the plight of the 5%, who constituted thousands of users experiencing significant frustration, often the most valuable customers with substantial data and heavy queries. The average effectively erased the very users it was meant to protect.

To address this, I switched to using percentiles throughout our monitoring systems. The 50th percentile (p50) revealed the typical experience, while the 99th percentile (p99) disclosed the worst experience endured by a significant portion of users. The disparity between p50 at 90ms and p99 at 6 seconds became glaringly evident: most requests were performing adequately, but a specific segment of users was facing catastrophic performance issues.

Relying solely on p99 across all traffic was insufficient, as it still represented an average of the worst-case scenarios. The true revelation came from slicing the data - breaking it down by endpoint, customer tier, region, and cache status. It became apparent that the slow tail was not a random occurrence but concentrated within a specific query pattern for customers with a high data volume.

This granular insight could not be gleaned from any aggregate data. The ability to dissect data along relevant dimensions is crucial in distinguishing between monitoring and observability. Monitoring involves constructing dashboards to answer questions we presume beforehand, while observability empowers us to inquire and investigate new questions - such as, "Is the system slow for large customers in Europe on a cold cache?" - during and after an incident, without necessitating code modifications.

It took me years to internalize this distinction in my professional approach. Above all, I implore organizations to cease reporting averages. Concealed within that seemingly harmless middle ground lies the truth - your most critical users are enduring your worst experience, and the average, by design, prevents its detection.