Kubernetes Controllers: The Cache-Staleness Bug at Scale
Key takeaways Structure reconcilers as ordered, independently testable steps, each reporting its own status condition, over one monolithic Reconcile function. Informer caches lag the API server by tens of milliseconds at scale; a stale read inside a MergeFrom patch can clobber a condition you set 49ms earlier. Fix status patching with an uncached reader inside RetryOnConflict ; Kubernetes v1.36…
The Kubernetes controllers pattern breaks a controller's reconcile loop into a series of ordered, independent steps, each reporting its own status condition. This approach keeps provisioning logic testable and allows for visible partial progress. However, informer caches can lag by tens of milliseconds, causing stale reads and overwriting of status conditions.
Kubernetes v1.36 introduced fixes to detect and skip behind-cache reconciles. A key to solving this issue is using a shared status-patching helper that centralizes conflict-safe read-then-patch operations. The helper reads through an uncached reader instead of the cache, re-fetches the latest object, mutates it, and patches it with a retry.RetryOnConflict loop.
This avoids race conditions between reconciling steps and ensures that unchanged state does not re-patch. Running the controller locally, which bypasses the shared cache, can help identify cache-related issues that persist in-cluster.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
