How NUMA Configuration Impacts Server Hardware Benchmarking

This article provides a comprehensive overview of how Non-Uniform Memory Access (NUMA) node configurations directly influence server hardware benchmarking. It covers the fundamentals of NUMA architecture, how memory locality and CPU interconnects affect latency and bandwidth, the role of PCIe device locality, and how BIOS settings and operating system configurations must be tuned to achieve accurate, repeatable, and maximum performance scores during hardware evaluations.

Understanding NUMA Architecture

In modern multi-socket servers—and even single-socket servers utilizing multi-chip module (MCM) designs like AMD EPYC or Intel Xeon Scalable processors—system resources are divided into NUMA nodes. Each NUMA node contains a specific set of CPU cores, local RAM channels, and PCIe lanes.

While any CPU core can access any memory address or PCIe device in the server, the speed of that access is "non-uniform." Accessing RAM attached directly to the CPU's own node (local access) is significantly faster than accessing RAM attached to a different node (remote access).

Memory Latency and Bandwidth Bottlenecks

The most direct impact of NUMA configuration on benchmarking occurs in memory-bound workloads.

  • Local Memory Access: When a benchmark runs on a NUMA node and uses only the RAM assigned to that node, data travels the shortest physical path. This yields the lowest memory latency and highest possible memory bandwidth.
  • Remote Memory Access: When a benchmark uses more memory than a single node can provide, or is improperly scheduled, the CPU must fetch data from another node. This requires traversing the CPU interconnect (such as Intel UPI or AMD Infinity Fabric). This extra "hop" increases latency by 30% to 100%+ and creates bandwidth bottlenecks on the interconnect, drastically lowering benchmark scores.

Thread Migration and OS Scheduling

Operating system schedulers constantly move threads across different CPU cores to balance thermal loads and utilization. During a benchmark, if the OS migrates a thread from Node 0 to Node 1, the thread's data remains in the physical RAM of Node 0.

Suddenly, a workload that was performing fast local memory reads is forced to perform slow remote memory reads. This causes high variance in benchmark results, making it difficult to reproduce scores consistently. Hardware testers must use CPU affinity (pinning) to lock benchmark threads to specific NUMA nodes to prevent this degradation.

PCIe and I/O Device Locality

Benchmarking storage (NVMe drives), networking (high-speed NICs), or compute accelerators (GPUs) is heavily influenced by NUMA. Every PCIe slot on a server motherboard is physically wired to a specific CPU socket and, consequently, a specific NUMA node.

If you run a storage benchmark like FIO to test an NVMe drive connected to Node 1, but the benchmark application executes on CPU cores in Node 0, all I/O data must travel across the CPU interconnect. This reduces maximum IOPS (Input/Output Operations Per Second), lowers throughput, and inflates storage latency. Accurate I/O benchmarking requires running the test software on the exact NUMA node to which the PCIe device is physically wired.

BIOS Settings: NPS and Sub-NUMA Clustering

Modern server BIOS menus allow administrators to alter how the hardware presents NUMA nodes to the operating system.

  • Nodes Per Socket (NPS) and Sub-NUMA Clustering (SNC): Settings like AMD’s NPS (NPS1, NPS2, NPS4) or Intel’s SNC divide a single physical processor into multiple smaller logical NUMA nodes.
  • Impact on Benchmarks: Configuring a system for multiple smaller nodes (e.g., NPS4) isolates memory controllers to their physically closest CPU cores. This optimizes the server for extreme low-latency and high-bandwidth benchmarks (like STREAM or High-Performance Computing workloads), provided the benchmark software is NUMA-aware. Conversely, setting the BIOS to a single monolithic node (e.g., NPS1) increases average memory latency slightly, which may lower peak synthetic benchmark scores, but often improves performance for unoptimized applications that do not understand NUMA boundaries.
  • Node Interleaving: Enabling Node Interleaving in the BIOS disables NUMA entirely by striping memory allocation across all nodes evenly. This destroys local memory advantages and usually results in lower peak benchmark scores, though it prevents worst-case scenarios where a single node's memory controller becomes entirely saturated.

Best Practices for NUMA Benchmarking

To get accurate and optimal results when benchmarking server hardware, follow these configuration rules:

  1. Balance Physical RAM: Populate DIMM slots symmetrically across all CPU sockets and memory channels. An unbalanced memory configuration forces uneven NUMA nodes and degrades bandwidth.
  2. Use numactl or Affinity Tools: On Linux, use the numactl command to bind both the benchmark process (--cpunodebind) and its memory allocation (--membind) to the same node. On Windows, use the /affinity flag with the start command or configure processor groups.
  3. Map PCIe Topology: Use tools like lstopo (from the hwloc package) to identify which NUMA node controls your target GPU, NIC, or NVMe drive. Bind your benchmarking tool to that specific node.
  4. Match BIOS Settings to the Workload: Use granular NUMA settings (SNC enabled, or NPS2/NPS4) for memory bandwidth and HPC benchmarks. Use broader settings (NPS1) for general database or virtualization benchmarks that require a large, unified pool of memory.