Trace log #3 -- Tracing one alert through Alert manager
The signal problem I was trying to understand how mature systems separate signal from noise when many alerts arrive at once. I had been working through a related question in TruthPipe, a project I built to represent uncertain data: once uncertainty is detected, how should the system deliver it? Alertmanager gave me a mature system for examining part of that delivery question. So I followed one…
Trace log #3 details the process of following an alert through the Alertmanager system. The Alertmanager handles alert management tasks, such as grouping, suppressing, routing, and delivering alerts. The model used in this system was already detailed, with components including the API, alert grouping and routing component (Dispatcher), the notification pipeline, and Gossip Settle.
An incoming alert is stored in two places: the Alert Provider, which holds active alerts, and the Silence Provider, which holds silence rules. The Dispatcher subscribes to new alerts, forms a group, and holds the group for a short wait. Deduplication checks are performed outside the Dispatcher on each receiver. After the group, the remaining steps occur in sequence: settling the cluster, suppressing a symptom alert, applying silence rules, choosing a route, waiting, skipping a sent notification, retrying, and sending.
The instrumentation map recorded ten logging points to trace the alert's journey, with the alert's fingerprint, group, receiver, integration, and flush included in each log line.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
