Our deploys were also our restart policy
At four on a Sunday morning an internal reconciliation service went down. All four pods were killed for memory inside twenty minutes of each other, came back, and were killed again a few hours later. The service had last been deployed in February. Seven months of continuous uptime, and it was the only thing in our estate that had any. The cause was a leak of about forty megabytes per pod per day.…
On a quiet Sunday morning, an internal reconciliation service experienced a failure. All four pods associated with the service crashed due to memory issues within a short span of twenty minutes. After a few hours, the service came back online, only to be terminated again. This service had been continuously running since February, making it the sole reliable component in the entire system.
The root cause of the issue was a memory leak of approximately forty megabytes per pod per day. The leak stemmed from an SDK client created for each request, which registered a listener on a static registry. However, this registry never received any unregistration, leading to an accumulation of one point two million listener objects over the course of seven months.
The problem originated from a library that had been in use since 2019, and it was a basic programming error. The team plotted the relationship between resident memory and pod age across the entire fleet, and they discovered that every service exhibited the same trend. The reason for this was that the median age of a pod in their clusters was under thirty hours.
Since the team deployed their services several times a day, a restart always cleared the leak, and no instance of any service had ever lived long enough to reach the memory limit. The only system that was functioning properly was the one that had never been modified, revealing the true nature of their code. The team rectified the client software, but they also implemented several other improvements.
They started alerting memory growth as a rate normalized by pod age, as this allowed for earlier detection of issues before they triggered a kill. Additionally, they introduced a soak environment where each service instance ran for thirty days against synthetic traffic, allowing them to graph the memory usage over time. Furthermore, they added pod age distribution to their platform dashboard, recognizing that a fleet with no pods older than two days could not provide meaningful insights about the system's behavior after thirty days.
The team had come to understand that frequent deployments had become a crucial reliability control measure, but they had never acknowledged it before. They realized that anything that masked a defect was a critical component of their infrastructure. If the only reason a system stayed up was due to constant restarts, a prolonged period of stability was a potential risk rather than a welcomed respite.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
