Urgent.News

What's breaking now, across thousands of outlets.

More in Tech

Pushgateway Heartbeat Gotcha: When ndots and NetworkPolicy Silently Eat Your Alerts

A CronJob exits 0, Pushgateway shows a heartbeat metric that hasn't updated in days, and Prometheus fires nothing. Every dashboard is green while the one thing the heartbeat was supposed to guarantee…

  • CronJob heartbeat metric fails to update for days, causing no alerts despite stopped job
  • Default ndots:5 DNS behavior and namespace-scoped NetworkPolicies compound monitoring failure
  • Wildcard DNS record and NetworkPolicy restrictions lead to silent packet drops and unnoticed issue

4 Non-obvious learnings from working with alerts

Alerting is a details game. Nobody gets paged by their intent — they get paged by the exact query they wrote. If you want to sleep through the night, the query has to mean what you think it means, and…

  • Throughput is age of oldest message, not rate floor
  • Catching tail requires per-unit deadline monitoring
  • Histograms limited by largest bucket present

More from Saturday 22 August →