How the Linux OOM Killer Works

The Linux Out of Memory (OOM) killer is an in-kernel safety mechanism designed to protect system stability when available RAM and swap space are completely exhausted. When the operating system faces an unrecoverable memory shortage, this mechanism steps in to evaluate active processes, select the most appropriate candidate based on a heuristic score, and terminate it with extreme prejudice. By terminating non-critical, memory-heavy workloads, the OOM killer frees up memory pages to prevent complete kernel panics and system lockups.

Memory Overcommit and the Need for the OOM Killer

Linux employs a memory allocation strategy known as overcommitting. When an application requests memory through system calls like malloc(), the kernel grants virtual address space without immediately backing it with physical RAM. Physical memory pages are only mapped when the process actually writes data to that allocated space.

This behavior maximizes resource utilization because applications rarely use all the memory they request. However, if multiple applications attempt to use their allocated memory simultaneously, total demand can exceed physical RAM and swap combined. When the kernel cannot fulfill these memory commitments through standard page reclamation or swapping, the OOM killer is triggered as a last-resort intervention.

How the Selection Algorithm Operates

When the kernel function out_of_memory() is invoked, it scans the system's process table to determine which task to terminate. The decision relies on an algorithm that calculates an oom_badness score for each eligible process.

The selection process evaluates several key factors:

• Memory Footprint: The primary metric is the proportion of physical memory (Resident Set Size or RSS) and swap space the process consumes. Larger consumers receive higher base scores.
• Privileged Execution: Processes running under the root user historically received a slight discount on their score, as terminating administrative tasks can destabilize the environment.
• Immunity of Critical Systems: Certain processes are strictly exempt from being killed. This includes PID 1 (system init, such as systemd) and core kernel threads. If PID 1 were terminated, the entire operating system would crash immediately.
• Process Lineage: The kernel generally avoids terminating an entire process hierarchy if killing a single child process can reclaim sufficient memory.

Manual Tuning with oom_score_adj

Administrators can influence the OOM killer's calculations through the /proc filesystem on a per-process basis using the /proc/[pid]/oom_score_adj file.

This value accepts integers ranging from -1000 to 1000:

• Setting a value of -1000 grants the process total immunity from the OOM killer.
• Setting a value of 1000 makes the process the primary target whenever an OOM condition occurs.
• Intermediate values proportionally increase or decrease the calculated badness score.

A read-only file, /proc/[pid]/oom_score, displays the final calculated value combining raw memory usage and the adjustment score.

Termination and Recovery

Once the kernel selects the process with the highest score, it immediately issues a SIGKILL (signal 9) to that task. Because SIGKILL cannot be caught, blocked, or ignored by the application, the process ceases execution immediately, and the kernel reclaims its mapped memory pages.

The event is recorded in the system log (accessible via dmesg or /var/log/syslog), containing details such as the killed process name, PID, memory usage statistics at the time of the event, and the total memory state of the host. If terminating the selected process does not release enough memory fast enough, the OOM killer will execute repeatedly until system stability is restored.