How RoCE Protocol Utilizes UDP

This article provides an overview of how the Remote Direct Memory Access over Converged Ethernet (RoCE) protocol integrates the User Datagram Protocol (UDP) to achieve high-performance, low-latency networking across routed environments. It examines the architectural shift from Layer 2 RoCEv1 to Layer 3 RoCEv2, the structure of UDP packet encapsulation, the critical function of UDP ports for traffic routing, and the implementation of Equal-Cost Multi-Path (ECMP) load balancing.

The Evolution from RoCEv1 to RoCEv2

The initial version of the protocol, RoCEv1, operated strictly at the Data Link layer (Layer 2) of the OSI model. It encapsulated InfiniBand transport headers directly inside standard Ethernet frames using a dedicated EtherType (0x8915). Because it lacked network-layer addressing, RoCEv1 was non-routable and confined strictly to single Layer 2 broadcast domains.

To overcome this limitation, the InfiniBand Trade Association (IBTA) introduced RoCEv2, also known as Routable RoCE (RRoCE). RoCEv2 replaces the direct Ethernet encapsulation by inserting standard IPv4 or IPv6 headers alongside a UDP transport header between the network layer and the InfiniBand payload.

UDP Packet Encapsulation Structure

In a RoCEv2 implementation, memory transfer operations bypass the host CPU and standard operating system network stacks. The RDMA-capable Network Interface Card (rNIC) generates the packet headers directly in hardware.

The resulting packet structure contains: 1. Ethernet Header: Standard Layer 2 addressing (MAC addresses). 2. IP Header: IPv4 or IPv6 addressing for Layer 3 inter-subnet routing. 3. UDP Header: The 8-byte transport header containing source and destination ports, length, and checksum fields. 4. InfiniBand Base Transport Header (BTH): Contains RDMA-specific control fields, such as the partition key, operation code, and destination Queue Pair (QP). 5. RDMA Payload: The actual application data being transferred. 6. Invariant CRC (ICRC) and Frame Check Sequence (FCS): Integrity verification fields.

The Role of UDP Port Numbers

RoCEv2 relies on UDP headers to interface seamlessly with standard data center switching hardware:

• Destination Port (Port 4791): The Internet Assigned Numbers Authority (IANA) designated UDP port 4791 specifically for RoCEv2 traffic. When receiving hardware (such as a top-of-rack switch or target rNIC) inspects the packet, port 4791 indicates that the UDP payload contains InfiniBand transport architecture data rather than standard socket payload.
• Source Port Calculation for ECMP: Modern data center networks utilize Equal-Cost Multi-Path (ECMP) routing to distribute traffic across parallel network links. Standard network switches parse the IP 5-tuple (Source IP, Destination IP, Protocol, Source Port, Destination Port) to calculate hashing buckets for link assignment. To avoid traffic bottlenecks caused by using a static port, the sending rNIC dynamically generates the UDP source port by hashing connection-specific metadata, such as the Queue Pair Number (QPN). This ensures distinct RDMA flows are distributed evenly across all available network paths without requiring deep packet inspection of the InfiniBand payload.

UDP Checksum Handling

In traditional networking, UDP includes an optional 16-bit checksum to detect data corruption across the IP and UDP headers and payload. In RoCEv2 deployments, calculating and validating standard UDP checksums can introduce unnecessary processing overhead.

Because RoCEv2 includes its own Invariant Cyclic Redundancy Check (ICRC) to protect the RDMA payload and headers across the entire path, the UDP checksum field is frequently set to 0x0000 (disabled) in IPv4 configurations. If IPv6 is used, where a UDP checksum is traditionally mandatory, specialized network hardware either offloads the computation to silicon or complies with RFC standards that permit zero-checksum modes for encapsulated tunnels.