{
  "id": 6882271,
  "title": "Nothing could start without the dependency we had filed as optional",
  "url": "https://urgent.news/2026/09/12/nothing-could-start-without-the-dependency-we-had-filed-as-optional",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-12T07:13:45.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/sergey_shinder_ab2d943365/nothing-could-start-without-the-dependency-we-had-filed-as-optional-d1o"
  },
  "original_language": "en",
  "account": "The configuration service was a tier three component, with no service level objective (SLO), no pager, and only a single replica. It was considered a soft dependency in the architecture document since clients cached their values and could continue functioning even if it was unavailable. However, the reality was starkly different during a routine node pool rotation at 09:40, when the configuration service went down, causing pods to be rescheduled across the fleet. A restarted pod had an empty in-memory cache, leading the client library to fetch configuration data during application startup. This process blocked and retried for two minutes before failing, resulting in readiness never being attained. Kubernetes then restarted the affected pods, but this triggered a cascade of failures, impacting services that were unrelated to configuration and were otherwise healthy. This incident highlighted the importance of recognizing hard dependencies, which are those dependencies a process cannot start without, regardless of their tier designation. The client was ultimately fixed, with the last known good configuration stored on disk and read at startup, along with defaults baked into the image for situations where there is no network. Consequently, the service was reclassified as tier one, with three replicas and a pager, due to its newfound criticality. This experience led to the adoption of a quarterly practice of blocking a dependency at the network level and restarting all services to uncover hidden dependencies, as relying solely on dependency diagrams could be misleading.",
  "summary": "Our configuration service was tier three. No SLO, no pager, a single replica for a long time, and a line in the architecture document describing it as a soft dependency because clients cache their values and can carry on if it is unavailable. That sentence was true about the steady state and completely false about the one moment that mattered. It went down at 09:40 during a routine node pool…",
  "key_points": [
    "Configuration service deemed soft dependency, but crucial for client startup",
    "Node pool rotation caused service outage, triggering cascade failures",
    "Service reclassified as tier one with added redundancy and monitoring"
  ],
  "editors_take": "The incident reveals that classifying dependencies as optional can be misleading, and reevaluating their criticality can lead to changes in service tier designation and increased redundancy.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}