Nothing could start without the dependency we had filed as optional

The configuration service was a tier three component, with no service level objective (SLO), no pager, and only a single replica. It was considered a soft dependency in the architecture document since clients cached their values and could continue functioning even if it was unavailable. However, the reality was starkly different during a routine node pool rotation at 09:40, when the configuration service went down, causing pods to be rescheduled across the fleet.

A restarted pod had an empty in-memory cache, leading the client library to fetch configuration data during application startup. This process blocked and retried for two minutes before failing, resulting in readiness never being attained. Kubernetes then restarted the affected pods, but this triggered a cascade of failures, impacting services that were unrelated to configuration and were otherwise healthy.

This incident highlighted the importance of recognizing hard dependencies, which are those dependencies a process cannot start without, regardless of their tier designation. The client was ultimately fixed, with the last known good configuration stored on disk and read at startup, along with defaults baked into the image for situations where there is no network.

Consequently, the service was reclassified as tier one, with three replicas and a pager, due to its newfound criticality. This experience led to the adoption of a quarterly practice of blocking a dependency at the network level and restarting all services to uncover hidden dependencies, as relying solely on dependency diagrams could be misleading.