Uncontrolled heap retention in Go: how to tell a leak from a healthy cache
Growing memory is not automatically a leak. In a real Go service, caches warm up, pools expand, buffers batch work, and connection managers hold state on purpose. If you only watch RSS or alloc_space , everything looks suspicious. The useful question is sharper: After GC, does retained heap keep climbing without a ceiling — and which call stacks are holding it? That is the problem godrain is…
Uncontrolled heap retention in Go can be challenging to distinguish from a healthy cache. While growing memory may not automatically indicate a leak, monitoring the retained heap after garbage collection (GC) is crucial. The godrain tool is designed to help with this process by analyzing the heap snapshots taken after GC and identifying suspicious call stacks holding onto memory.
The main idea behind godrain is to sample the heap with GC=1, aggregate the inuse_space by call stack, drop a warm-up window, and then analyze the slope, plateau, and any allowed list or load correlation to classify the growth as ok, expected, suspect, or a leak. Allowlists are important, as without them, a growing cache during warm-up might be incorrectly flagged as a leak.
Godrain provides valuable insights into a live Go service, helping to differentiate between intentional allocations, such as caches, pools, and buffers, and unintended growth. For example, when a global map is being filled by a goroutine, godrain can attribute the leak to that specific goroutine. In contrast, a bounded LRU cache that should plateau will show a different growth pattern, leading to a different verdict.
The tool works by sampling the heap at regular intervals, aggregating the inuse_space by call stack, dropping the warm-up window, and then scoring the slope, plateau, and applying any allowed list or load correlation. The final verdict includes the top stacks contributing to the retained heap.
To use godrain effectively, it is essential to expose the service's pprof endpoint privately (e.g., on 127.0.0.1) and warm the process under realistic traffic before running the watch. By running godrain watch with appropriate parameters, such as warmup, duration, interval, and load metrics, you can identify the source of retained memory more accurately.
However, it's important to note that godrain is not a replacement for interactive debugging tools like go tool pprof, a static analyzer, or a proof of a leak. It is a heuristic designed to automate the process of identifying potential leaks in a service under steady load. False positives and false negatives can occur, such as when the watch window is shorter than the cache fill time or when intentional growth is not allowlisted. Longer soaks and thorough analysis can help minimize these issues.
The primary purpose of godrain is to automate the tedious process of monitoring heap growth over time and pinpointing the call stacks responsible for retaining memory. It is not intended to replace manual debugging or static analysis, but rather to provide a more efficient way to investigate potential leaks in Go services.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
