How Linux Handles NUMA Architecture

Non-Uniform Memory Access (NUMA) is a multiprocessing architecture where memory access times depend on the memory's location relative to a specific processor. In modern multi-socket and high-core-count systems, the Linux operating system plays a vital role in optimizing performance across these disparate memory topologies. This article explores how the Linux kernel organizes physical hardware into NUMA nodes, manages memory allocation policies, coordinates task scheduling, and dynamically migrates memory through Automatic NUMA Balancing to minimize latency and maximize throughput.

NUMA Hardware Topology in Linux

In a NUMA system, processors are grouped with dedicated local memory into distinct hardware units called NUMA nodes. Accessing local memory attached to the socket running a process provides low latency and high bandwidth. Accessing remote memory—memory physically wired to another socket via an interconnect like Intel UPI or AMD Infinity Fabric—incurs higher latency and reduced bandwidth.

The Linux kernel discovers this physical topology during boot time through the Advanced Configuration and Power Interface (ACPI) tables, specifically the System Resource Affinity Table (SRAT) and System Locality Information Table (SLIT). The SLIT defines a relative cost matrix between nodes, allowing the kernel to quantify the penalty of accessing remote memory.

Kernel Abstraction: Nodes, Zones, and Pages

To manage NUMA effectively, the Linux kernel structures physical memory hierarchically:

  1. Nodes (pg_data_t): Each physical NUMA node is represented in the kernel by a pg_data_t structure. This structure tracks the node’s total memory, state, and active CPU list.
  2. Zones: Within each node, memory is split into zones (such as ZONE_DMA32 and ZONE_NORMAL) to handle physical addressing constraints.
  3. Pages: Individual page frames are allocated via the kernel's buddy allocator, which is instantiated per-node to avoid cross-socket lock contention.

Memory Allocation Policies

By default, Linux implements a first-touch allocation policy. When an application requests memory using system calls like malloc(), virtual memory addresses are mapped, but physical pages are not allocated immediately. The physical allocation occurs on the first memory access (page fault), and the kernel allocates memory from the NUMA node where the thread triggering the fault is currently executing.

Linux also exposes several fine-grained allocation policies that can be configured system-wide or per-process:

These policies can be set programmatically via libnuma or the set_mempolicy() system call, and externally via the numactl command-line utility.

NUMA-Aware Task Scheduling

Memory locality depends heavily on the CPU scheduler. The Linux Completely Fair Scheduler (CFS) incorporates NUMA awareness by grouping CPUs into scheduling domains.

The scheduler aims to:

If a task is moved to a remote socket to prevent CPU starvation, its memory remains on the original node, turning fast local memory accesses into slower remote accesses.

Automatic NUMA Balancing (AutoNUMA)

To resolve the mismatch between where a task runs and where its memory resides, Linux includes Automatic NUMA Balancing (kernel.numa_balancing).

AutoNUMA operates through three main mechanisms:

  1. NUMA Hinting Faults: The kernel periodically marks memory pages as non-accessible. When the running thread accesses them, a low-overhead "hinting fault" is generated.
  2. Access Tracking: The fault handler records which CPU and node accessed the page.
  3. Dynamic Migration: If a page is consistently accessed by a CPU on a remote node, the kernel migrates the page to that node's local memory. Conversely, the scheduler may migrate the task to the node where the majority of its memory already exists.

Inspection and Diagnostics

Linux provides administrators and developers with tools to monitor NUMA interactions directly: