{
  "id": 19436204,
  "title": "Your Server Died at 3 AM: Reading the Linux OOM Killer's Report (and Stopping It)",
  "url": "https://urgent.news/2026/11/11/your-server-died-at-3-am-reading-the-linux-oom-killers-report-and",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-11T03:16:54.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/jun_liu_ca219b73a8f1489b5/your-server-died-at-3-am-reading-the-linux-oom-killers-report-and-stopping-it-e7h"
  },
  "original_language": "en",
  "account": "Every system administrator encounters the dreaded 3 AM incident of a server suddenly ceasing to respond. The first clue is often a dmesg log message stating an \"Out of memory: Killed process.\" The Linux OOM (Out of Memory) killer is responsible for this decision, and understanding how it works can significantly reduce the mystery and impact of these incidents.\n\nThe OOM killer is triggered when the system runs out of memory and must make a decision to kill a process. It computes a \"badness\" score for each process based primarily on its resident anonymous memory (RSS) and kills the one with the highest score. The key points to remember are that the victim is rarely the process that actually caused the memory shortage, and that swap space is not a long-term solution to memory issues.\n\nTo better understand and manage the OOM killer, it's crucial to read the kernel logs. These can be accessed via dmesg or journalctl, which provide detailed information about the process being killed, including its memory usage and a score known as oom_score_adj. This score indicates whether the process has been intentionally nudged up or down in priority.\n\nTwo main controls can be used to manage the OOM killer's behavior: OOMScoreAdjust and MemoryHigh/MemoryMax settings. The OOMScoreAdjust setting, available through systemd, allows you to manually adjust the OOM score of a specific process, making it either more likely to be killed or less likely. A value of -900 makes a process extremely resistant to being killed, while a value of 500 makes it a prime candidate for immediate termination.\n\nThe MemoryHigh and MemoryMax settings, defined in a cgroup configuration, provide a more proactive approach. MemoryHigh acts as a soft limit, triggering aggressive memory reclaim and throttling before the system reaches the hard MemoryMax limit. This allows services to degrade gracefully rather than being abruptly killed. MemoryMax sets a firm limit, and crossing this threshold results in an OOM event specific to the cgroup, ensuring that only the offending process is affected while the rest of the system remains operational.\n\nProperly configuring these settings can prevent entire hosts from crashing due to a single runaway process. By making the failure contained and predictable, system administrators can provide clearer, more actionable postmortems and improve overall system stability.",
  "summary": "It happens to every ops person eventually. The on-call page fires at 03:12: a customer-facing node has stopped answering, yet the process list looks normal and CPU is low. You SSH in, and the first line of dmesg tells you what actually happened: Out of memory: Killed process 4123 (java) total-vm:31457280kB, anon-rss:28901376kB, file-rss:1024kB The kernel did not crash. It did not run out of disk.…",
  "key_points": [
    "Linux OOM killer terminates highest memory using process at 3 AM",
    "OOMScoreAdjust controls process priority for OOM killer",
    "MemoryHigh/MemoryMax settings prevent system-wide OOM crashes"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}