{
  "id": 6276129,
  "title": "My Auto-Remediation Bot Fixed the Wrong Service. Here's What That Taught Me About Real SRE.",
  "url": "https://urgent.news/2026/09/08/my-auto-remediation-bot-fixed-the-wrong-service-heres-what-that",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-08T12:02:25.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/lucas_ferreira_/my-auto-remediation-bot-fixed-the-wrong-service-heres-what-that-taught-me-about-real-sre-23ad"
  },
  "original_language": "en",
  "account": "I constructed a system designed to detect failures, determine appropriate actions, execute them autonomously, and document the entire process without any human intervention. The first test of this system under real conditions resulted in multiple errors. This was not a confession of failure but rather the most valuable aspect of the entire project, and that is what I wish to discuss.\n\nTo provide context, I developed a portfolio platform that emulates a simple e-commerce system, consisting of three microservices connected in a real dependency chain (orders → inventory → notifications) and running on Kubernetes. On top of this, I layered the standard stack used by any serious SRE team: Prometheus and Grafana for observability, formal SLOs with error budgets, Chaos Mesh to intentionally inject failures, and the most ambitious component - a Python controller that autonomously reacts when something goes wrong. The core concept: close the full loop. Detect → decide → act → learn. This is not just about monitoring and waiting for a human to become available.\n\nGame Day 005 marked the first time the auto-remediation controller faced a genuine failure - not a synthetic test, but an actual chaos breaking an actual service. The plan was to sustain an outage in the estoque-service (inventory), trigger an alert through Alertmanager, and observe the controller's autonomous reaction. After waiting, I discovered three issues, one after another.\n\nIssue 1 involved the alert not reaching the controller. While the Alertmanager and Prometheus were configured correctly, the alert was not propagated downstream. The root cause was found to be the Prometheus Operator automatically adding a namespace filter to every alert route - my rules lacked this label, causing the alert to fall into an empty receiver without triggering any error logs. The fix involved adding the missing label.\n\nIssue 2 arose when the alert arrived, but the controller acted on the wrong service. It was programmed to only consider the service that triggered the alert (pedidos-service, the orders service) and had no concept that the root cause was actually in a dependency (estoque-service). Consequently, it restarted the healthy orders service while the actual problem continued unabated.\n\nIssue 3 revealed that the controller falsely claimed recovery in just 48 seconds. This happened because when the Prometheus query returned no data initially (which is normal right after any action due to the scrape interval), the code defaulted to checking if the pod was in a 'Ready' state via the Kubernetes API. A 'Ready' pod is not the same as a healthy service, but that's what the system treated as success. The cause was that the controller treated a ready pod as recovery, which is incorrect.",
  "summary": "I built a system that detects failure, decides what to do, acts on its own, and documents everything afterward — no human in the loop. The first time I tested it against a real failure, it got things wrong. Not once. Three times. This isn't a confession of failure. It's the most valuable part of the whole project, and that's what I want to talk about. Quick context I built a portfolio platform…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}