How Does RDMA Over InfiniBand Bypass the Kernel?

Remote Direct Memory Access over InfiniBand minimizes network latency by allowing a network adapter to read and write directly to application memory without involving the operating system kernel. In traditional networking, every packet must pass through kernel-space protocol stacks, undergo multiple memory copies, and trigger context switches that consume CPU cycles and add microseconds of delay. By moving data transfer operations into user space and offloading transport logic onto specialized hardware, RDMA over InfiniBand achieves sub-microsecond latencies and high throughput for classical high-performance computing workloads.

The Bottleneck of Traditional Kernel Networking

In a conventional TCP/IP stack, sending or receiving data requires the application to make a system call, such as send() or recv(). This triggers a context switch from user mode to kernel mode, which flushes CPU pipelines and pollutes caches. Once inside the kernel, the operating system allocates socket buffers, copies data between user space and kernel space, and processes transport-layer headers.

On the receiving side, the network interface card (NIC) raises a hardware interrupt, forcing the CPU to pause its current work, execute an interrupt service routine, and copy the incoming payload from the kernel buffer into the application's memory. Under heavy workloads, these interrupts and memory copies create a severe bottleneck, limiting both latency and scalability.

How InfiniBand Enables Kernel Bypass

InfiniBand replaces the traditional NIC with a Host Channel Adapter (HCA) that implements the entire transport protocol in hardware. Instead of relying on the operating system to manage sockets and buffers, the HCA communicates directly with user-space applications through a mechanism known as OS bypass.

During initialization, the application registers a region of its virtual memory with the HCA via the InfiniBand Verbs API. The operating system pins these memory pages in physical RAM so they cannot be swapped out to disk, and it provides the HCA with the physical-to-virtual address translation tables. Once registration is complete, the kernel steps out of the data path entirely.

Queue Pairs and User-Space Doorbells

Data transfers in InfiniBand are managed through Queue Pairs (QPs), each consisting of a Send Queue and a Receive Queue, alongside a Completion Queue (CQ). These queues reside in user-space memory, allowing the application to post work requests directly without making system calls.

To initiate a transfer, the application writes a Work Queue Element (WQE) into its Send Queue and rings a hardware "doorbell" by writing to a memory-mapped I/O register on the HCA. The HCA reads the WQE via Direct Memory Access (DMA), fetches the payload directly from the registered application buffer, packetizes it, and transmits it over the InfiniBand fabric. When the operation finishes, the HCA writes a completion entry to the Completion Queue, which the application can poll in user space without waiting for a hardware interrupt.

One-Sided vs. Two-Sided Operations

InfiniBand RDMA supports both two-sided and one-sided communication models:

  • Two-Sided (Send/Receive): Similar to traditional messaging, the sender posts a Send request and the receiver must have a matching Receive buffer posted in advance. Both sides are aware of the transfer.
  • One-Sided (RDMA Read/Write): The initiator specifies both the local memory address and the remote virtual address (along with a remote access key). The remote CPU is completely unaware of the transfer; the remote HCA handles the memory access autonomously.

One-sided operations are particularly effective for minimizing latency because they eliminate synchronization overhead between the communicating processes.

Quantifying the Latency Advantage

Total end-to-end network latency \(L_{\text{total}}\) in a classical system can be modeled as the sum of serialization delay, propagation delay, and host processing overhead:

\[L_{\text{total}} = L_{\text{host}} + L_{\text{wire}} + L_{\text{switch}}\]

In traditional Ethernet/TCP setups, \(L_{\text{host}}\)—comprising system calls, context switches, and buffer copies—often accounts for \(10\text{--}50\,\mu\text{s}\) or more. By bypassing the kernel and offloading transport to the HCA, RDMA over InfiniBand reduces \(L_{\text{host}}\) to a fraction of a microsecond, bringing total point-to-point latency down to roughly \(1\text{--}2\,\mu\text{s}\) across modern InfiniBand fabrics.

Applications in Classical Computing

Because of its ultra-low latency and minimal CPU overhead, RDMA over InfiniBand is the standard interconnect for supercomputers, distributed databases, and large-scale machine learning clusters. By freeing the CPU from network processing, classical compute nodes can dedicate nearly all of their cycles to computation rather than data movement.