Your Server Died at 3 AM: Reading the Linux OOM Killer's Report (and Stopping It)

Every system administrator encounters the dreaded 3 AM incident of a server suddenly ceasing to respond. The first clue is often a dmesg log message stating an "Out of memory: Killed process." The Linux OOM (Out of Memory) killer is responsible for this decision, and understanding how it works can significantly reduce the mystery and impact of these incidents.

The OOM killer is triggered when the system runs out of memory and must make a decision to kill a process. It computes a "badness" score for each process based primarily on its resident anonymous memory (RSS) and kills the one with the highest score. The key points to remember are that the victim is rarely the process that actually caused the memory shortage, and that swap space is not a long-term solution to memory issues.

To better understand and manage the OOM killer, it's crucial to read the kernel logs. These can be accessed via dmesg or journalctl, which provide detailed information about the process being killed, including its memory usage and a score known as oom_score_adj. This score indicates whether the process has been intentionally nudged up or down in priority.

Two main controls can be used to manage the OOM killer's behavior: OOMScoreAdjust and MemoryHigh/MemoryMax settings. The OOMScoreAdjust setting, available through systemd, allows you to manually adjust the OOM score of a specific process, making it either more likely to be killed or less likely. A value of -900 makes a process extremely resistant to being killed, while a value of 500 makes it a prime candidate for immediate termination.

The MemoryHigh and MemoryMax settings, defined in a cgroup configuration, provide a more proactive approach. MemoryHigh acts as a soft limit, triggering aggressive memory reclaim and throttling before the system reaches the hard MemoryMax limit. This allows services to degrade gracefully rather than being abruptly killed. MemoryMax sets a firm limit, and crossing this threshold results in an OOM event specific to the cgroup, ensuring that only the offending process is affected while the rest of the system remains operational.

Properly configuring these settings can prevent entire hosts from crashing due to a single runaway process. By making the failure contained and predictable, system administrators can provide clearer, more actionable postmortems and improve overall system stability.