Understanding Zone Reclaim Mode in Linux NUMA

Zone reclaim mode in a Non-Uniform Memory Access (NUMA) Linux system controls whether the kernel reclaims local memory or allocates from remote nodes when a specific NUMA node runs low on free pages. This setting directly balances the trade-off between memory access latency and CPU overhead, defining how aggressively the operating system prioritizes NUMA locality over immediate allocation speed.

The Role of Zone Reclaim in NUMA Systems

In a NUMA architecture, physical memory is partitioned across multiple nodes, each directly attached to a specific CPU socket. A CPU accessing its local memory experiences minimal latency, whereas accessing memory on a remote socket over interconnect buses (such as Intel UPI or AMD Infinity Fabric) incurs a noticeable latency penalty.

When a process requests memory on a node that lacks free pages, the Linux kernel faces two choices:

1. Reclaim local memory: Evict cached data, write back dirty pages, or swap to free space on the local node to preserve memory locality.
2. Allocate remotely: Immediately satisfy the request using available free memory from a neighbor node, avoiding the delay of page reclamation at the cost of higher subsequent access latencies.

The vm.zone_reclaim_mode sysctl parameter dictates this decision.

Configuration Values and Behaviors

The vm.zone_reclaim_mode parameter is configured via a bitmask that specifies which operations the kernel is permitted to perform to free local pages:

• 0 (Disabled): The kernel does not attempt aggressive local reclamation. If the local zone cannot satisfy the allocation, the kernel immediately allocates pages from remote NUMA nodes. This is the modern default behavior.
• 1 (Bit 0): Enables zone reclaim. The kernel scans and frees clean, unmapped page cache entries locally before falling back to remote allocation.
• 2 (Bit 1): Allows writing dirty pages to disk during zone reclaim to free additional local memory.
• 4 (Bit 2): Allows swapping anonymous memory out to disk to reclaim space on the local node.

These flags can be combined. For example, a value of 3 enables clean page cache reclamation and dirty page writebacks, while 7 enables all reclamation methods including swapping.

Performance Implications

Enabling zone reclaim mode creates a significant risk of execution stalls. While accessing local memory reduces memory bus latency, the process of scanning zones, shrinking caches, and writing to disk consumes significant CPU cycles and introduces direct reclaim latency.

• Database and In-Memory Workloads: For workloads like PostgreSQL, MySQL, and Redis, enabling zone reclaim mode frequently causes sudden, massive latency spikes. If local nodes fill up, the system stalls execution threads to evict caches instead of consuming available RAM on adjacent nodes. For these systems, vm.zone_reclaim_mode should be set to 0.
• High-Performance Computing (HPC): In compute-bound parallel workloads with working sets strictly bounded to fit within a single NUMA node, setting zone reclaim to 1 ensures that threads do not inadvertently pull in remote memory, preserving peak memory bandwidth and consistent latency.

Inspecting and Modifying Zone Reclaim

To check the current zone reclaim configuration:

sysctl vm.zone_reclaim_mode

To set the value dynamically without rebooting (for example, disabling it):

sysctl -w vm.zone_reclaim_mode=0

To make the setting permanent, append vm.zone_reclaim_mode = 0 to /etc/sysctl.conf or an appropriate file in /etc/sysctl.d/.