Ubuntu OOM Killer: Managing Memory Exhaustion

The Out of Memory (OOM) Killer is a safeguard mechanism built into the Linux kernel that activates when an Ubuntu system faces critical memory depletion. This article explains the fundamental purpose of the OOM Killer, how memory overcommit causes resource crises, the mathematical heuristic the kernel uses to select and terminate sacrificial processes, and how administrators can monitor and tune this behavior to maintain system availability.

The Purpose of the OOM Killer

In Ubuntu, the kernel operates with a feature known as memory overcommit. Because applications often request more virtual memory than they actively use, the kernel routinely promises more memory than the combined total of physical RAM and swap space. While this maximizes resource utilization under normal workloads, it creates a risk: if multiple applications suddenly demand their full allocations simultaneously, the operating system runs completely out of allocatable physical pages.

When physical RAM and swap are entirely exhausted and the kernel cannot satisfy a memory allocation request, the system enters an Out of Memory state. To prevent an unrecoverable kernel panic or a complete system freeze, the kernel invokes the OOM Killer. Its sole job is to sacrifice one or more processes to reclaim memory instantly and keep the operating system running.

How the OOM Killer Selects a Process

The OOM Killer does not terminate processes at random. Instead, it computes a "badness" score for every running process, assigning each an oom_score ranging from 0 to 1000. The process with the highest score is targeted first.

The calculation evaluates several criteria:

  • Memory Footprint: Processes utilizing the largest percentage of physical RAM and swap are penalized most heavily, as terminating them recovers the most memory immediately.
  • Root vs. User Privileges: Critical system services and processes owned by the root user receive slight score reductions to prevent destabilizing essential infrastructure.
  • Process Lifetime: Newly spawned, rapidly expanding processes are prioritized over long-running foundational daemons.
  • Hardware Access: Processes performing direct hardware access are generally avoided to prevent device corruption.

Once the highest-scoring target is identified, the OOM Killer issues a non-catchable SIGKILL (signal 9). The kernel immediately reclaims all memory pages assigned to that process.

Modern Ubuntu and systemd-oomd

Recent Ubuntu releases (including Ubuntu 22.04 LTS and later) introduce systemd-oomd, a userspace out-of-memory manager that complements the kernel-level OOM Killer.

While the kernel OOM Killer acts as an emergency last line of defense when allocations fail entirely, systemd-oomd acts proactively. Using control groups (cgroups v2) and Pressure Stall Information (PSI), systemd-oomd detects severe memory pressure before the kernel runs out of pages entirely. It terminates misbehaving application slices or cgroups earlier in the exhaustion cycle, resulting in cleaner recoveries and fewer system freezes.

Monitoring and Adjusting OOM Behavior

Administrators can inspect OOM events by querying kernel logs using dmesg or journalctl:

dmesg -T | grep -i oom
journalctl -k | grep -i "killed process"

The output reveals the exact process terminated, its memory usage at the time of death, and the final oom_score.

If a mission-critical service (such as a database) must never be killed, administrators can manually bias the score using the oom_score_adj interface located at /proc/[PID]/oom_score_adj. Values range from -1000 (completely immune to the OOM Killer) to 1000 (highest termination priority):

echo -1000 | sudo tee /proc/<PID>/oom_score_adj

For persistent adjustments, service definitions in systemd can include the OOMScoreAdjust= directive within the [Service] unit block.