{
  "id": 6869384,
  "title": "We tagged EKS workers like app servers — and Prometheus started scraping telegraf on nodes that never had it",
  "url": "https://urgent.news/2026/09/12/we-tagged-eks-workers-like-app-servers-and-prometheus-started",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-12T05:25:01.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/mridul_it_is/we-tagged-eks-workers-like-app-servers-and-prometheus-started-scraping-telegraf-on-nodes-that-2n59"
  },
  "original_language": "en",
  "account": "The Telegraf Down alerts began accumulating on a Thursday morning, initially appearing concerning. The first thought was a fleet-wide agent failure, but it turned out to be even more complex. The issue stemmed from a governance work done three days earlier, which had unintended consequences on monitoring. The tagging work, seemingly harmless, led to Prometheus scrapping telegraf on nodes that had never had it before. The problem arose when EKS worker nodes, after receiving governance tags, were included in the EC2 service discovery pool, allowing Prometheus to scrape them using the standard port 9273. However, Kubernetes workers do not run a host telegraf process on this port, resulting in the metrics being collected from in-cluster DaemonSets or cAdvisor instead. This led to five EKS worker targets showing as permanently up=0 and triggering a Telegraf Down alert, despite there being no telegraf process on these nodes to begin with. The root causes were diverse, including a misconfigured AL2023 ASG instance, a disabled Kafka exporter, and a redis host with issues. The EKS issue was the easiest to address, requiring a simple Prometheus relabel configuration change, while the AL2023 hosts required more extensive fixes.",
  "summary": "The Telegraf Down alerts started piling up on a Thursday morning, and at first glance it looked bad. Nine targets, all critical, all under the same rule name. My first instinct was a fleet-wide agent failure — something in the monitoring stack had broken overnight and we were about to page half the platform team for a problem that didn't exist. It wasn't that. It was worse in a quieter way: one…",
  "key_points": [
    "Governance tags caused Prometheus to scrape telegraf on EKS nodes without it",
    "EKS workers included in EC2 service discovery pool for Prometheus scraping",
    "Simple Prometheus relabel change fixed the EKS issue"
  ],
  "editors_take": "Misapplying governance tags to EKS workers changes how Prometheus treats them, mistakenly targeting nodes without telegraf and triggering false alerts, which reveals flaws in service discovery and monitoring configurations.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}