The Phantom OOM: How a Silent Session Leak Brought Down Our Microservice Every 48 Hours

Our telemetry microservice would inevitably crash every 48 hours due to an Out Of Memory (OOM) condition. The issue lay hidden in the ORM session lifecycle. The background tasks instantiated long-lived database sessions that were never properly closed, leading to a slow, linear increase in memory usage and exhausted connections to the PostgreSQL database.

Investigating the problem, we discovered that background task functions were creating global or long-lived sessions without explicit commit or close handling. When exceptions occurred early, the session object and its identity map cache remained active, trapping network sockets in idle states that the garbage collector could not reclaim. Over thousands of iterations, this gradually depleted memory until Kubernetes terminated the pod.

To resolve the issue, we replaced manual session handling with strict context managers to ensure sessions were always closed and transactions rolled back. Background tasks now used generator-based context managers, guaranteeing that sessions, transactions, and socket connections were properly released. We also configured explicit pool recycling in SQLAlchemy to prevent stale connections from accumulating.

Implementing these changes had a dramatic impact. Memory usage stabilized at a predictable 350MB, eliminating the OOM crashes. The service became continuously available with zero memory-related restarts over several months, and PostgreSQL connections remained healthy without unnecessary idle states.