How Do GPUs Communicate Over NVLink in AWS P4d and P5?

When multiple GPUs inside a single Amazon EC2 P4d or P5 instance communicate over NVLink and NVSwitch, they exchange data directly across a high-bandwidth, low-latency fabric without routing traffic through the host CPU, system memory, or PCIe switches. This hardware-level peer-to-peer interconnect creates a unified memory space across all eight local GPUs, allowing collective communication operations and direct memory transfers to execute at hundreds of gigabytes per second per GPU while keeping host resources free for other tasks.

The Architecture of Intra-Node GPU Interconnects

In traditional multi-GPU systems, communication between accelerators relies on the Peripheral Component Interconnect Express (PCIe) bus. When one GPU needs to send tensor data to another, the data often travels through PCIe root complexes or switches, creating a severe bandwidth bottleneck. Even with PCIe Gen4 or Gen5, the maximum bidirectional bandwidth per GPU is limited to \(64\text{ GB/s}\) or \(128\text{ GB/s}\), respectively.

To eliminate this bottleneck in high-performance computing and deep learning workloads, Amazon EC2 P4d and P5 instances utilize NVIDIA NVLink and NVSwitch technologies:

  • NVLink is a wire-level serial communication protocol developed by NVIDIA that enables direct, high-speed point-to-point connections between GPUs.
  • NVSwitch is a physical switching chip located on the server baseboard (such as the NVIDIA HGX A100 or HGX H100 board) that connects multiple NVLink lanes into a fully connected, non-blocking crossbar fabric.

Instead of only connecting adjacent GPUs in a ring or mesh topology, NVSwitch allows any GPU inside the instance to communicate with any other GPU simultaneously at full NVLink bandwidth.

Hardware Specifications in EC2 P4d and P5 Instances

Both the P4d and P5 instance families house 8 GPUs per instance, interconnected via NVSwitch, though their underlying generations differ significantly in throughput and capabilities.

Amazon EC2 P4d Instances (NVIDIA A100)

An ec2 p4d.24xlarge instance contains eight NVIDIA A100 Tensor Core GPUs connected via third-generation NVLink and second-generation NVSwitch chips:

  • Links per GPU: 12 third-generation NVLink connections
  • Bidirectional Bandwidth per Link: \(50\text{ GB/s}\)
  • Total Bidirectional Bandwidth per GPU: \(600\text{ GB/s}\) (\(300\text{ GB/s}\) ingress and \(300\text{ GB/s}\) egress)
  • Aggregate Instance NVLink Bandwidth: \(4.8\text{ TB/s}\)

Amazon EC2 P5 Instances (NVIDIA H100)

An ec2 p5.48xlarge instance contains eight NVIDIA H100 Tensor Core GPUs connected via fourth-generation NVLink and third-generation NVSwitch chips:

  • Links per GPU: 18 fourth-generation NVLink connections
  • Bidirectional Bandwidth per Link: \(50\text{ GB/s}\)
  • Total Bidirectional Bandwidth per GPU: \(900\text{ GB/s}\) (\(450\text{ GB/s}\) ingress and \(450\text{ GB/s}\) egress)
  • Aggregate Instance NVLink Bandwidth: \(7.2\text{ TB/s}\)
  • In-Network Compute: Third-generation NVSwitch introduces NVIDIA Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) engines directly inside the NVSwitch silicon.

Step-by-Step Mechanics of GPU-to-GPU Communication

When a distributed workload—such as training a large language model using PyTorch and the NVIDIA Collective Communications Library (NCCL)—triggers an intra-node transfer, several coordinated steps occur beneath the software layer.

1. Topology Discovery and Peer Access Initialization

When the CUDA runtime and NCCL initialize on a P4d or P5 instance, they query the NVIDIA driver to discover the physical interconnect topology. Detecting the NVSwitch fabric, the runtime enables Peer-to-Peer (P2P) memory access across all 8 GPUs.

Through Unified Virtual Addressing (UVA), the High Bandwidth Memory (HBM2e on P4d, HBM3 on P5) of all eight GPUs is mapped into a single, contiguous virtual address space. A kernel executing on GPU 0 can reference a memory pointer residing physically on GPU 7 just as it would reference its own local memory.

2. Bypassing the Host CPU and PCIe Bus

Once peer access is established, data transfers between GPUs completely bypass the host x86 CPUs, system RAM, and PCIe root complex. There is no need to stage data in host-pinned memory.

When a transfer is initiated—either via an explicit cudaMemcpyPeerAsync call or via a GPU kernel reading/writing remote pointers directly—the GPU's internal memory controller translates the virtual memory address into a physical NVLink packet target.

3. Routing Across the NVSwitch Crossbar

The originating GPU serializes the memory transaction into flits (flow control digits) and transmits them across its NVLink lanes to the onboard NVSwitch chips.

Because the NVSwitch fabric forms a full crossbar topology, traffic from all 8 GPUs is striped across multiple NVSwitch ASICs simultaneously. If GPU 0 sends data to GPU 1 while GPU 2 sends data to GPU 3, neither transfer contends for bandwidth with the other. Every GPU can transmit at its maximum unidirectional rate (\(300\text{ GB/s}\) on P4d, \(450\text{ GB/s}\) on P5) and receive at the same rate concurrently.

4. Remote Memory Access and Cache Coherency

On the receiving GPU, the incoming NVLink packets are reassembled and routed directly to the target GPU's High Bandwidth Memory controller or L2 cache. GPU kernels can even perform atomic operations (such as atomicAdd) directly on the memory of a remote peer GPU over NVLink, enabling fine-grained synchronization without host intervention.

In distributed machine learning, GPUs frequently execute collective operations like AllReduce, AllGather, ReduceScatter, and AllToAll to synchronize gradients and activations.

Ring and Tree Algorithms on P4d (NVSwitch Gen 2)

On a P4d instance, NCCL constructs logical rings or double binary trees over the physical NVSwitch fabric. During an AllReduce operation across \(N = 8\) GPUs with a data payload of size \(S\), a traditional ring algorithm requires each GPU to send and receive:

\[\text{Data Transferred per GPU} = 2 \times \frac{N - 1}{N} \times S\]

With \(N = 8\), each GPU transmits \(1.75 \times S\) bytes over NVLink, and the GPUs themselves use their Streaming Multiprocessors (SMs) to compute the mathematical sum of incoming gradient chunks before forwarding them to the next GPU in the logical schedule.

Hardware-Accelerated Collectives on P5 (NVSwitch Gen 3)

On a P5 instance, the third-generation NVSwitch chips include dedicated Arithmetic Logic Units (ALUs) that support NVLink SHARP (in-switch reductions). When NCCL executes an AllReduce on a P5 instance using NVLink SHARP:

  1. Instead of passing chunks from GPU to GPU in a ring, a GPU issues a multicast or reduction request directly to the NVSwitch fabric.
  2. The NVSwitch chips read the corresponding memory addresses from all 8 GPUs concurrently.
  3. The ALUs inside the NVSwitch hardware compute the sum (or min/max) of the tensors across all 8 GPUs inside the switch itself.
  4. The NVSwitch broadcasts the final reduced result back to all 8 GPUs simultaneously using hardware multicast.

This reduces the required NVLink traffic per GPU from \(2 \frac{N-1}{N} S\) down to approximately \(S\), frees up GPU Streaming Multiprocessors for compute tasks rather than communication kernels, and boosts the effective AllReduce algorithm bandwidth on a P5 instance from roughly \(350\text{ GB/s}\) to nearly \(480\text{ GB/s}\) per GPU.

Interaction with Elastic Fabric Adapter (EFA) in Multi-Node Clusters

While NVLink and NVSwitch handle all intra-node communication inside a single P4d or P5 instance, large-scale training clusters span multiple instances connected via AWS Elastic Fabric Adapter (EFA).

Inside a multi-node cluster, NCCL uses a hierarchical communication strategy:

  1. Intra-node phase: GPUs within the same instance aggregate or scatter data at ultra-high speed (\(600\text{ GB/s}\) or \(900\text{ GB/s}\)) over NVLink and NVSwitch.
  2. Inter-node phase: Only the necessary cross-instance data is routed out of the GPUs over PCIe Gen4 (P4d) or PCIe Gen5 (P5) via GPUDirect RDMA directly to the EFA network interface cards (\(400\text{ Gbps}\) on P4d, \(3200\text{ Gbps}\) across 32 EFA devices on P5).

This division of labor allows model parallelism techniques—such as Tensor Parallelism, which requires heavy layer-by-layer synchronization—to stay strictly within the single-instance NVSwitch domain, while Data Parallelism and Pipeline Parallelism operate across instances over EFA.