{
  "id": 8432983,
  "title": "The configuration we changed in June arrived one pod at a time in August",
  "url": "https://urgent.news/2026/09/19/the-configuration-we-changed-in-june-arrived-one-pod-at-a-time-in",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-19T10:00:52.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/sergey_shinder_ab2d943365/the-configuration-we-changed-in-june-arrived-one-pod-at-a-time-in-august-3d7g"
  },
  "original_language": "en",
  "account": "In June, the team decided to change a value related to job duration from ten minutes to two minutes in one of their systems. They edited a ConfigMap, which was then deployed as environment variables in the container at startup. The change was considered complete by the team, as the ConfigMap remained the same and did not trigger a deployment.\n\nHowever, the situation took a different turn in August when the node pool was upgraded. The pods were taken down one by one, and six pods returned with the updated two-minute lease while the remaining eighteen continued with the original ten-minute lease. This led to a situation where a send that took three minutes would sit inside the old lease and outside the new one. When the mail provider was slow, a job already in progress would be handed to another pod, resulting in the customer receiving the same message twice.\n\nTo address this issue, the team implemented an annotation in each deployment that holds a hash of its configuration. Editing a ConfigMap would now change the pod template, triggering a rollout independently. Additionally, each service exposes the loaded hash as a metric, and an alert is triggered if more than one value is reported for a rollout period.\n\nUltimately, the team realized that there was no clear indication of the distance between the applied change and the change that was already running. This incident highlighted the need for better tracking and monitoring of configuration changes to avoid similar issues in the future.",
  "summary": "A support ticket said our dispatch emails were going out twice. Not always, not to any customer we could pick out, and never while we watched. Twenty four pods, one image, one deployment, one ConfigMap, and six of those pods were running a value we had changed in June. The value is a lease: how long a worker holds a job before the queue decides it has died and gives the job to someone else. In…",
  "key_points": [
    "Configuration change from ten to two minutes occurred in June",
    "Node pool upgrade in August caused mixed lease durations",
    "Annotation and metric implementation resolved configuration issue"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}