Which EC2 Instances and OS Support EFA OS-Bypass via Libfabric?

Amazon EC2 Elastic Fabric Adapter (EFA) provides OS-bypass capabilities through the Libfabric communications library across high-performance Linux operating systems and numerous compute-optimized, memory-optimized, accelerated computing, and general-purpose EC2 instance families. By routing packets directly from user-space applications to the custom-built Nitro network hardware, EFA eliminates kernel context-switching overhead, significantly reducing network latency and maximizing throughput for tightly coupled High Performance Computing (HPC) and distributed Machine Learning (ML) workloads. This article details the architecture of EFA OS-bypass via Libfabric, lists the supported Amazon EC2 instance families, outlines compatible operating systems, and explains how the software stack enables low-latency communication at scale.

How EFA OS-Bypass and Libfabric Work Together

Traditional TCP/IP networking requires applications to invoke system calls that transition execution from user space to the operating system kernel. The kernel processes network headers, manages socket buffers, and handles interrupts before passing data to the network interface card (NIC). While reliable for general-purpose workloads, this path introduces microsecond-level latency jitter and consumes CPU cycles that could otherwise be dedicated to computation.

Elastic Fabric Adapter overcomes this bottleneck using an architecture known as OS-bypass. Through the OpenFabrics Interfaces (OFI) Libfabric API, user-space libraries communicate directly with the EFA hardware queues managed by the AWS Nitro System:

  1. User-Space Queue Management: When an application initializes communication via MPI (such as Open MPI or Intel MPI) or collective communication libraries (such as AWS OFI NCCL for PyTorch and TensorFlow), Libfabric allocates send, receive, and completion queues directly in user-space memory.
  2. Direct Memory Access (DMA) and RDMA: The EFA provider (ofi_efa) within Libfabric registers memory regions with the EFA device. Data transfers occur via DMA and Remote Direct Memory Access (RDMA) read/write operations without copying buffers through the OS kernel.
  3. Scalable Reliable Datagram (SRD): EFA relies on the AWS-engineered Scalable Reliable Datagram (SRD) protocol in the Nitro hardware. Unlike traditional InfiniBand, which struggles with congestion over Equal-Cost Multi-Path (ECMP) routing, SRD sprays packets across multiple network paths simultaneously and handles out-of-order delivery and retransmissions directly in hardware.

The effective latency \(L_{\text{total}}\) in an OS-bypass model eliminates the kernel traversal time \(T_{\text{kernel}}\), leaving only the user-space library serialization \(T_{\text{lib}}\), PCIe traversal \(T_{\text{PCIe}}\), and physical network propagation \(T_{\text{fabric}}\):

\[L_{\text{total}} = T_{\text{lib}} + 2 \cdot T_{\text{PCIe}} + T_{\text{fabric}}\]

Supported Operating Systems for EFA OS-Bypass

Note that while all current-generation EC2 instances support the ENA (Elastic Network Adapter) driver for standard TCP/IP networking across almost any OS, OS-bypass via Libfabric on EFA requires specific operating system kernel drivers (efa.ko via rdma-core) and user-space libraries packaged in the official AWS EFA installer.

Supported Linux Distributions

AWS provides full OS-bypass support via the EFA installer on the following 64-bit (x86_64 and arm64 Graviton) Linux operating systems:

  • Amazon Linux: Amazon Linux 2023 (AL2023) and Amazon Linux 2 (AL2)
  • Ubuntu: Ubuntu 20.04 LTS, Ubuntu 22.04 LTS, and Ubuntu 24.04 LTS
  • Red Hat Enterprise Linux (RHEL): RHEL 8 and RHEL 9 (and compatible derivatives)
  • Rocky Linux: Rocky Linux 8 and Rocky Linux 9
  • AlmaLinux: AlmaLinux 8 and AlmaLinux 9
  • SUSE Linux Enterprise Server (SLES): SLES 15 (SP4 and later recommended for modern HPC stacks)
  • CentOS: CentOS 7 and CentOS Stream 8/9 (legacy support in earlier EFA installer releases)

Windows Server Support Limitations

While EFA devices can be attached to certain Windows Server instances for standard IP networking, OS-bypass via Libfabric is not supported on Windows operating systems. Distributed workloads requiring EFA OS-bypass and SRD capabilities must run on one of the supported Linux distributions listed above, either natively on the host or inside privileged Linux containers (such as Docker, Singularity/Apptainer, or AWS Batch/EKS pods with /dev/infiniband/uverbs* device pass-through).

Supported Amazon EC2 Instance Families

EFA OS-bypass support is available primarily on large and bare-metal (metal) instance sizes, though many newer generations (such as 7th and 8th generation instances) also support EFA on smaller virtualized sizes down to 4xlarge or 8xlarge.

1. Accelerated Computing Instances (ML & GPU HPC)

Accelerated computing instances frequently pair EFA with multiple GPUs or custom AI accelerators, using multiple EFA attachments per instance to achieve up to \(3200\text{ Gbps}\) of aggregate network bandwidth:

  • P-Family (NVIDIA GPU Accelerated): p3dn.24xlarge, p4d.24xlarge, p4de.24xlarge, p5.48xlarge, p5e.48xlarge, p5en.48xlarge
  • Trn-Family (AWS Trainium): trn1.32xlarge, trn1n.32xlarge, trn2.48xlarge
  • Inf-Family (AWS Inferentia): inf2.24xlarge, inf2.48xlarge
  • G-Family (Graphics & ML Inference/Training): g4dn.8xlarge, g4dn.12xlarge, g4dn.16xlarge, g4dn.metal, g5.8xlarge, g5.12xlarge, g5.16xlarge, g5.24xlarge, g5.48xlarge, g6.8xlarge, g6.12xlarge, g6.16xlarge, g6.24xlarge, g6.48xlarge, g6e.8xlarge, g6e.12xlarge, g6e.16xlarge, g6e.24xlarge, g6e.48xlarge, dl1.24xlarge, dl2q.24xlarge

2. High Performance Computing (HPC) Instances

Purpose-built for tightly coupled scientific simulations, weather forecasting, computational fluid dynamics (CFD), and finite element analysis:

  • Hpc6-Family: hpc6a.48xlarge, hpc6id.32xlarge
  • Hpc7-Family: hpc7a.12xlarge, hpc7a.24xlarge, hpc7a.48xlarge, hpc7a.96xlarge, hpc7g.4xlarge, hpc7g.8xlarge, hpc7g.16xlarge

3. Compute-Optimized Instances

Compute-optimized instances use EFA for CPU-based MPI workloads and high-throughput distributed processing:

  • C5-Family: c5n.9xlarge, c5n.18xlarge, c5n.metal
  • C6-Family: c6i.32xlarge, c6i.metal, c6id.32xlarge, c6id.metal, c6in.16xlarge, c6in.24xlarge, c6in.32xlarge, c6in.metal, c6a.32xlarge, c6a.48xlarge, c6a.metal, c6gn.16xlarge
  • C7-Family: c7i.24xlarge, c7i.48xlarge, c7i.metal-24xl, c7i.metal-48xl, c7a.24xlarge, c7a.32xlarge, c7a.48xlarge, c7a.metal-48xl, c7g.16xlarge, c7g.metal, c7gd.16xlarge, c7gd.metal, c7gn.8xlarge, c7gn.12xlarge, c7gn.16xlarge, c7gn.metal
  • C8-Family: c8g.24xlarge, c8g.48xlarge, c8g.metal-24xl, c8g.metal-48xl

4. Memory-Optimized Instances

Ideal for in-memory databases, electronic design automation (EDA), and large-memory scientific modeling:

  • R5-Family: r5n.8xlarge through r5n.24xlarge, r5n.metal, r5dn.8xlarge through r5dn.24xlarge, r5dn.metal
  • R6-Family: r6i.24xlarge, r6i.32xlarge, r6i.metal, r6id.24xlarge, r6id.32xlarge, r6id.metal, r6in.16xlarge, r6in.24xlarge, r6in.32xlarge, r6in.metal, r6idn.16xlarge, r6idn.24xlarge, r6idn.32xlarge, r6idn.metal, r6a.32xlarge, r6a.48xlarge, r6a.metal
  • R7-Family: r7i.24xlarge, r7i.48xlarge, r7i.metal-24xl, r7i.metal-48xl, r7iz.16xlarge, r7iz.32xlarge, r7iz.metal-16xl, r7iz.metal-32xl, r7a.24xlarge, r7a.32xlarge, r7a.48xlarge, r7a.metal-48xl, r7g.16xlarge, r7g.metal, r7gd.16xlarge, r7gd.metal
  • R8-Family: r8g.24xlarge, r8g.48xlarge, r8g.metal-24xl, r8g.metal-48xl
  • X2-Family (High Memory): x2idn.24xlarge, x2idn.32xlarge, x2idn.metal, x2iedn.24xlarge, x2iedn.32xlarge, x2iedn.metal, x2iezn.8xlarge, x2iezn.12xlarge, x2iezn.metal
  • U7i-Family (Ultra High Memory): u7i-12tb.224xlarge, u7in-16tb.224xlarge, u7in-24tb.224xlarge, u7in-32tb.224xlarge

5. General-Purpose and Storage-Optimized Instances

Used for balanced distributed applications, scale-out file systems (such as Lustre or WekaIO), and data-intensive analytics clusters:

  • M5-Family: m5n.8xlarge through m5n.24xlarge, m5n.metal, m5dn.8xlarge through m5dn.24xlarge, m5dn.metal, m5zn.6xlarge, m5zn.12xlarge, m5zn.metal
  • M6-Family: m6i.24xlarge, m6i.32xlarge, m6i.metal, m6id.24xlarge, m6id.32xlarge, m6id.metal, m6in.16xlarge, m6in.24xlarge, m6in.32xlarge, m6in.metal, m6idn.16xlarge, m6idn.24xlarge, m6idn.32xlarge, m6idn.metal, m6a.32xlarge, m6a.48xlarge, m6a.metal
  • M7-Family: m7i.24xlarge, m7i.48xlarge, m7i.metal-24xl, m7i.metal-48xl, m7a.24xlarge, m7a.32xlarge, m7a.48xlarge, m7a.metal-48xl, m7g.16xlarge, m7g.metal, m7gd.16xlarge, m7gd.metal
  • M8-Family: m8g.24xlarge, m8g.48xlarge, m8g.metal-24xl, m8g.metal-48xl
  • I-Family (Storage-Optimized): i3en.6xlarge, i3en.12xlarge, i3en.24xlarge, i3en.metal, i4i.16xlarge, i4i.32xlarge, i4i.metal, i4g.16xlarge, im4gn.16xlarge

Verifying EFA OS-Bypass Support and Configuration

Once an EFA-enabled EC2 instance is launched with a supported Linux OS and an EFA network interface attached, you can verify that the kernel module, rdma-core user-space libraries, and Libfabric EFA provider are operational.

Checking the Libfabric EFA Provider

Run the fi_info utility included with the AWS EFA installer to query the active EFA Libfabric providers:

fi_info -p efa

A properly configured instance returns output showing the efa provider supporting both FI_EP_RDM (Reliable Datagram endpoints, which utilize the hardware SRD protocol and support RDMA operations) and FI_EP_DGRAM (Datagram endpoints):

provider: efa
    fabric: efa
    domain: rdmap0s31-rdm
    version: 121.0
    type: FI_EP_RDM
    protocol: FI_PROTO_EFA
provider: efa
    fabric: efa
    domain: rdmap0s31-dgrm
    version: 121.0
    type: FI_EP_DGRAM
    protocol: FI_PROTO_EFA

Querying EFA Support via AWS CLI

Because AWS continuously expands EFA support to newly released EC2 instance families and sizes, you can dynamically query the exact list of EFA-supported instances in your target AWS Region using the AWS CLI:

aws ec2 describe-instance-types \
    --filters Name=network-info.efa-supported,Values=true \
    --query "InstanceTypes[*].[InstanceType, NetworkInfo.EfaInfo.MaximumEfaInterfaces, NetworkInfo.NetworkPerformance]" \
    --output table

Key Requirements for Enabling OS-Bypass in Production

To ensure your workloads actively use the OS-bypass path rather than falling back to standard TCP over ENA, verify the following configuration requirements:

  • Security Group Rules: An EFA-enabled security group must allow all inbound and outbound traffic (all protocols and ports) to and from itself. Because SRD operates directly over the Nitro fabric outside standard TCP/UDP port numbers, restrictive port rules will block OS-bypass traffic.
  • Placement Groups: Launch instances inside a Cluster Placement Group to minimize physical distance and switch hops between nodes.
  • Memory Locking Limits (ulimit -l): Because OS-bypass pins user-space memory pages for DMA transfers, the operating system must allow locked memory (memlock) to be set to unlimited in /etc/security/limits.conf or container runtime configurations.
  • HugePages and GPUDirect RDMA: On GPU instances (p4d, p5, g5, g6e) and Trainium instances (trn1, trn2), install the aws-ofi-nccl plugin and NVIDIA nvidia-peermem (or Neuron runtime drivers) alongside Libfabric so that EFA can perform GPUDirect RDMA transfers directly between GPU HBM memory and the network card without staging through host system RAM.