The Drift Problem That Breaks GitOps at Scale

The key issue with GitOps at scale is not the sync engine itself, but the problem of "drift" - the differences that arise in the live cluster after Git indicates everything is in sync. Drift can be caused by several sources: controllers that add defaults or mutate resources, deploy pipelines that patch live manifests for canaries or rollbacks, and operators that manage subresources and continuously rewrite fields. These all create a state that diverges from Git, requiring different responses.

One of the biggest mistakes teams make is assuming self-heal will fix everything. Enabling self-heal for everything causes problems when rollback tools need to quickly patch live deployments, as self-heal will quickly revert those changes, making rollbacks ineffective. A better approach is to disable self-heal for namespaces where live state is legitimately patched by tools, and use a diff job that catches drift without reverting it, alerting on the specific namespaces.

A useful diff job can be scheduled from CronJob or CI to check for orphaned or progressing resources and report any drift, comparing live state to the target state via the API. The output can then be piped to Slack for quick notification to the right owners. Meanwhile, teams should declare which fields operators legitimately own by ignoring those differences in the Application manifest, treating them as debt items with owners.

Ultimately, the deploy pipeline is the real fix for most pipeline drift - instead of direct patches, commit to an environment overlay and let sync handle the process. This makes rollbacks simpler (git revert) and creates a clear audit trail. One audited break-glass path that pauses sync before patches are applied can handle emergencies without ad-hoc patches breaking the system. Setting up the drift detection early, capping ignoreDifferences entries, and documenting who can touch live state are all key recommendations.