Our deploy script kept every value except the ones the new chart added
In August a node in our production cluster came under memory pressure and the kubelet evicted every checkout pod on it, while pods of other services on the same node, using more memory, stayed put. Checkout lost a third of its capacity for six minutes. The events explained it in one word: the evicted pods were BestEffort. They had no resource requests at all. They were supposed to. Since April…
In August, a memory pressure issue in a production cluster node caused the kubelet to evict checkout pods, leaving the service with only a third of its capacity for six minutes. The cause was traced back to BestEffort pods lacking resource requests. Since April, the shared service chart had implemented default requests and limits for all services, causing every service repository to update to version five.
As a result, twenty-three out of thirty-one services in production lacked these values. The deploy script, written three years ago, used --reuse-values to maintain hand-set initial values, unintentionally preserving the chart's defaults. Consequently, production releases from 2023 inherited these default values and continued to run without explicit resource requests.
To address this, the team now deploys with explicit environment-specific values files and rejects pods without memory requests in production namespaces. A new deployment process includes collecting existing settings, checking for differences with production, and displaying helm diffs in pull requests.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
