Attaching Evidence to Alerts: an Enrichment Sidecar for Alertmanager
Every on-call engineer knows the ritual. A page lands in Slack: the 5xx rate on a service is above the threshold. The message tells you that something is wrong and nothing about what. So you open Kibana or Loki in another tab, set the time window to the last few minutes, filter by the service, […]
A sidecar service, running alongside Alertmanager, improves alert notifications by attaching relevant logs directly to the alert message. This sidecar addresses the limitations of Alertmanager's template system, which can only use labels and annotations already attached by Prometheus. Previous attempts at solution included a bot posting logs to Slack, but this approach led to the paging path becoming the bot and rendering it unreliable.
The final solution uses a second receiver alongside Alertmanager, posting alerts to Slack while simultaneously sending them to the sidecar service. When the sidecar cannot find matching logs, it sends a message indicating the lack of relevant log entries. The service keeps the alert notifications unaltered, allowing the team to focus on diagnosis without additional steps to find relevant logs.
Rewritten by urgent.news; may contain errors — check the original.
Sources: devops-hd5ff.sevalla.app · devops.com
This story
This is one outlet's version. Read the fullest account.