Our latency alert summed away the bucket label and never returned a value

Our checkout service employs a latency alert: notify someone if the ninety-ninth percentile remains above two seconds for ten minutes. Last spring, the alert was rewritten. The outdated version averaged a percentile across pods, which was deemed incorrect. The updated version followed the documentation's guidance to sum histogram buckets first and then take the quantile.

After review, merging, and loading, Prometheus listed the rule as healthy. However, in September, a slow query in the basket service caused checkout's p99 latency to spike to six seconds for most of an hour. Customers became aware of this issue. The dashboard's latency panel clearly displayed this problem. Despite this, no one was paged.

The rule specified: histogram_quantile(0.99, sum by (service) (rate(http_request_duration_seconds_bucket[5m]))). A Prometheus histogram consists of counters, each representing a bucket, differentiated by the le label, which indicates the bucket's upper bound. The histogram_quantile function requires this label to correctly identify the buckets.

The sum by (service) clause kept only the service label, causing the function to combine all buckets into a single series. Consequently, the le label was lost before histogram_quantile could process the data. The query did not fail in such situations; instead, it returned nothing, and an alert with no output was not triggered. The dashboard panel was created separately using sum by (le), which is why it could still detect the problem.

For five months, this rule was evaluated every thirty seconds, but it consistently produced no result. The rules page displayed the rule as healthy throughout, as health only indicates that the query ran without errors. The solution was a single word: sum by (service, le). Additionally, tests were added. Each alerting rule now includes a promtool test file with synthetic series, ensuring that at least one scenario triggers the alert.

Continuous integration (CI) runs these tests on every change, and a nightly job checks each alert's expression, removing the final comparison and running the remaining query against production. If an expression returns no series, the team responsible for the alert is notified. On the first night, this process identified two more alerts, including one with a misspelled metric name.

An alert that cannot fire appears similar to an alert with nothing to report. To prevent such issues, every rule now shows evidence of firing at least once. This improvement was implemented by Sergey Shinder.