Benchmarking PCIe P2P Speeds in Multi-Card Workstations

This article explains the methodology and tools used to test PCIe peer-to-peer (P2P) transfer speeds in multi-card workstation configurations. It covers the initial verification of hardware topology, the industry-standard benchmarking software utilized for GPUs and accelerators, the step-by-step execution of unidirectional and bidirectional bandwidth tests, and how hardware evaluators interpret these results against theoretical PCIe generation limits.

1. Verifying Hardware Topology

Before running speed tests, benchmarkers must map the physical and logical layout of the motherboard. PCIe P2P allows expansion cards, most commonly GPUs, to read and write data directly to each other's onboard memory over the PCIe bus without routing the data through the system RAM.

Testers use command-line utilities, such as nvidia-smi topo -m for NVIDIA hardware or rocm-smi --showtopo for AMD hardware, to view the connection matrix. The highest P2P speeds occur when cards communicate through a dedicated onboard PCIe switch (P2P via PIX/PXB). If the data must pass through the CPU's PCIe root complex (PHB) or cross between two different CPUs via interconnects like AMD Infinity Fabric or Intel UPI (SYS), P2P performance degrades, or the feature may be blocked entirely by the system architecture.

2. Industry-Standard Benchmarking Tools

Hardware testers rely on vendor-specific software development kits (SDKs) to isolate and measure raw PCIe P2P performance, predominantly using Linux environments to minimize operating system overhead.

  • NVIDIA CUDA Samples: The primary tool for NVIDIA GPUs is the p2pBandwidthLatencyTest binary, which testers compile directly from the NVIDIA CUDA Toolkit. It specifically isolates and measures data transfer rates and latency between all available GPU pairs.
  • AMD ROCm Bandwidth Test: For AMD workstation and data center cards, testers run the rocm-bandwidth-test utility. Using specific flags, this tool validates P2P accessibility and benchmarks direct copy speeds between devices.
  • NCCL and RCCL Tests: To test how P2P speeds affect real-world machine learning and high-performance computing workloads, benchmarkers compile NVIDIA Collective Communications Library (NCCL) or AMD RCCL tests. Running binaries like all_reduce_perf stresses the PCIe bus across three or more cards simultaneously.

3. The Testing Methodology

When a benchmarker runs a P2P test suite, the software executes a strict sequence of memory operations to gather accurate metrics:

  1. P2P Access Check: The software queries the hardware driver to confirm if Device A has permission to directly access the memory address space of Device B.
  2. Memory Allocation: The benchmark allocates a fixed buffer size (typically scaling from a few kilobytes up to several gigabytes) in the local memory (VRAM) of each card.
  3. Baseline Test (P2P Disabled): The software forces data to travel from Card 1, through the CPU and system RAM, and then to Card 2. This establishes a baseline speed.
  4. Unidirectional P2P Test: P2P is enabled, and data is pushed directly from Card 1 to Card 2 using Direct Memory Access (DMA). A high-precision timer records the duration of the transfer over multiple loops to calculate an average speed in GB/s.
  5. Bidirectional P2P Test: Both cards simultaneously send and receive data to each other. This saturates the full-duplex capability of the PCIe lanes, effectively doubling the total bandwidth utilization.
  6. Latency Measurement: The benchmark sends an empty or microscopic payload (such as 4 bytes) back and forth between the cards thousands of times to measure the communication delay, which is recorded in microseconds (\(\mu\)s).

4. Interpreting the Results

Benchmark results are evaluated in Gigabytes per second (GB/s) and compared against the theoretical limits of the workstation's PCIe generation and lane width. Because the PCIe protocol uses encoding schemes (such as 128b/130b) and requires packet headers, real-world P2P speeds never reach the raw theoretical maximum.

  • PCIe 3.0 x16: Out of a 15.75 GB/s theoretical limit, a healthy unidirectional P2P transfer yields roughly 12 to 13 GB/s.
  • PCIe 4.0 x16: Out of a 31.5 GB/s theoretical limit, testers expect to see effective P2P speeds between 25 and 28 GB/s.
  • PCIe 5.0 x16: Out of a 63 GB/s theoretical limit, properly functioning P2P transfers achieve roughly 50 to 56 GB/s.

In bidirectional tests, these numbers should nearly double (e.g., reaching up to 52 GB/s on PCIe 4.0 x16). If the benchmark results are significantly lower than these expected ranges, or if the bidirectional speed is the exact same as the unidirectional speed, the tester knows a bottleneck exists. This usually indicates that the motherboard is splitting the x16 lanes into x8/x8, the CPU root complex is throttling the P2P packets, or P2P has silently failed and fallen back to routing data through the slower system RAM.