Three Silent Failures Between You and Sidecar Injection

Running the command `kubectl label deployment my-app pandocore.io/inject=enabled` mistakenly modifies the Deployment's top-level metadata.labels rather than the spec.template.metadata.labels. This seemingly small error leads to three silent failures that prevent sidecar injection from working correctly.

First, `kubectl` reports the label change, giving a false sense of success. Second, since the pod template remains unchanged, the Deployment controller does not initiate a rollout, resulting in no new pods being created. Third, even if pods were to be recreated, they would lack the necessary label and thus never trigger the mutating admission webhook responsible for injecting sidecars. Consequently, users may see all pods running with a status of 1/1, but without the desired sidecar container.

The issue arises because the label is applied at the wrong level, leading to confusion when troubleshooting. The straightforward instruction to label the Deployment appears correct, and the command executes without errors, making the failure less apparent. The fix involves applying the label directly to the pod template using `kubectl patch deployment my-app --type merge -p '{"spec":{"template":{"metadata":{"labels":{"pandocore.io/inject":"enabled"}}}}}`.

This approach triggers a rollout, creating new pods with the correct label and allowing them to pass through the webhook for sidecar injection. If the label is set differently and the cause of failure is unclear, running `kubectl rollout restart deployment my-app` ensures fresh pods are deployed, with the sidecar container loaded.

Verifying with `kubectl get pods -l pandocore.io/inject=enabled` confirms the presence of both containers in the running pods. Ultimately, the root cause lies in applying the label to the Deployment's metadata instead of the pod template, highlighting the importance of understanding the underlying structure when managing Kubernetes resources.