How an Unpinned Base Image Tag Let V8 Size Its Heap Off the Host, Not the Pod

On a Tuesday, pods within the checkout-api deployment began undergoing multiple OOMKilling incidents. Traffic was stable, and no new releases had been deployed. The initial suspicion was a memory leak introduced the previous day, taking a day to accumulate enough pressure to impact pod performance. The git log for the past 48 hours revealed just one commit, a routine lockfile refresh unrelated to application code changes.

Traffic rates remained consistent with the previous Tuesday, indicating the issue was not related to increased load. Another theory suggested a slow leak crossing a threshold, but the heapUsed metric remained constant at 140MB, while the rss grew steadily past 480MB, exceeding the pod's 512MB limit. V8's heap accounting did not reflect this growth.

The root cause was traced to a change in the base image used by the deployment. The Dockerfile initially used a floating tag, FROM node:18-alpine, but the nightly CI rebuild process fetched the latest available version, causing V8 to size its heap incorrectly based on the host's total memory instead of the pod's 512MB limit. This discrepancy allowed V8 to commit excessive memory, leading to OOMKilling.

To resolve the issue, the base image should be pinned to a specific digest, ensuring consistent builds and preventing silent changes. Additionally, the heap ceiling should be explicitly set to the pod's memory limit, preventing V8 from misinterpreting the available resources. Finally, the CI job should be updated to fail if the base image line is not pinned, preventing similar issues in the future.