4 Non-obvious learnings from working with alerts

Working with alerts requires attention to detail. The exact query written determines when an alert triggers. An example demonstrated this well: an alert on message processing count was set as a throughput signal for a file processor. The colleague didn't realize that one message contains up to ten URLs, so the metric tracked was wrong. This led to tracking the wrong number and downstream effects.

There are four key lessons to learn from this:

1. Throughput is age-of-oldest-message, not a rate floor. A floor alert for fewer than N items in M minutes can't distinguish between a wedged system and idle time. Age of the oldest queued message resolves this, as it rises if work is queued but not moving. This requires a model of your workload to determine what's normal.

2. Catching the tail is important. Queue age only shows that work isn't draining. To know why, you need a per-unit deadline to see if any single job exceeds N minutes. Many exceedances indicate a tail issue, while few exceedances mean the main workstream is healthy. Ratios need a volume gate, not just a non-zero check. A 99% success rate sounds generous, but with sparse workloads, one failure can breach the SLO.

Including a volume gate in the ratio or percentile-shaped SLI ensures the alert only fires when there's enough work to mean something.

3. You can only see as far as your largest bucket. Histograms are useful for tracking job durations, but they only work at boundaries that actually exist. If no jobs took longer than a certain time, a bucket for that time won't exist, and the alert won't fire. This can lead to false confidence that everything is fine when it's not.