Designing Alerts That Explain an Incident
TL;DR An alert is a message a monitoring system sends when a measurement crosses a threshold, and its worth shows up in the decision it lets someone take in the next ten minutes. Google's SRE Workbook recommends paging when a service burns 2% of its error budget in one hour, which is a burn rate of 14.4, and when it burns 5% in six hours; the same table sends 10% in three days to a ticket queue.…
An alert is a message sent by a monitoring system when a measurement exceeds a predetermined threshold. Its effectiveness is measured by the decision it enables someone to make within the next ten minutes. Google's SRE Workbook suggests paging when a service consumes 2% of its error budget in one hour, or 5% in six hours. A ten-minute window on a 99.9% objective triggers an alert for an event that consumes 0.02% of the monthly budget.
Grouping, inhibition, silences, and mute timings help prevent one faulty dependency from generating numerous notifications. A standard alert should include the symptom, the affected scope, the time window, and the runbook, as well as the person on call.
For a small team, the starting configuration includes one page-level threshold, one ticket-level threshold, and a weekly review. Measuring whether the pages earn their purpose involves reviewing the actual alerts that have been triggered. An alert must contain information about the user's current situation, the extent of the service's impact, and where to find instructions. A notification that answers these questions leads to action; one that merely reports a metric name results in a search.
Prometheus' Alertmanager documentation explains how the component processes alerts it receives: it removes duplicates, groups them, and routes them to a receiver like email, PagerDuty, or OpsGenie. Deduplication and routing apply to existing alerts, so the design decisions occur upstream in the alerting rules that determine when the system has a message to convey.
The Alertmanager's own documentation outlines grouping, inhibition, silences, and mute timings, which are mechanisms to prevent a single broken dependency from generating a hundred notifications.
By using a burn-rate framing, the threshold in the alert describes the rate at which the service consumes its error budget relative to the objective. Two percent of a 30-day budget spent in one hour equals a burn rate of 14.4, while five percent in six hours equals 6. Both thresholds should be part of a paging path. Ten percent in three days represents a burn rate of 1, and this threshold is placed in a ticket queue, suitable for a slow leak that a regular workday can handle.
A multiwindow alert checks a short window alongside a longer one, with one twelfth of the longer window's duration as a guideline. This helps prevent an alert from continuously firing once the errors have ceased.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
