We tagged EKS workers like app servers — and Prometheus started scraping telegraf on nodes that never had it
The Telegraf Down alerts started piling up on a Thursday morning, and at first glance it looked bad. Nine targets, all critical, all under the same rule name. My first instinct was a fleet-wide agent failure — something in the monitoring stack had broken overnight and we were about to page half the platform team for a problem that didn't exist. It wasn't that. It was worse in a quieter way: one…
The Telegraf Down alerts began accumulating on a Thursday morning, initially appearing concerning. The first thought was a fleet-wide agent failure, but it turned out to be even more complex. The issue stemmed from a governance work done three days earlier, which had unintended consequences on monitoring. The tagging work, seemingly harmless, led to Prometheus scrapping telegraf on nodes that had never had it before.
The problem arose when EKS worker nodes, after receiving governance tags, were included in the EC2 service discovery pool, allowing Prometheus to scrape them using the standard port 9273. However, Kubernetes workers do not run a host telegraf process on this port, resulting in the metrics being collected from in-cluster DaemonSets or cAdvisor instead.
This led to five EKS worker targets showing as permanently up=0 and triggering a Telegraf Down alert, despite there being no telegraf process on these nodes to begin with. The root causes were diverse, including a misconfigured AL2023 ASG instance, a disabled Kafka exporter, and a redis host with issues. The EKS issue was the easiest to address, requiring a simple Prometheus relabel configuration change, while the AL2023 hosts required more extensive fixes.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
