How Does AWS ENA Deliver High EC2 Network Throughput?

The Elastic Network Adapter driver achieves up to hundreds of gigabits per second of throughput on Amazon EC2 by combining custom AWS Nitro hardware offload with a scalable, multi-queue software architecture. Rather than relying on legacy software-emulated network interfaces, the Elastic Network Adapter exposes a lightweight PCI Express virtual interface directly to the guest operating system. By utilizing independent transmit and receive rings pinned to dedicated virtual CPUs, hardware-assisted checksum and segmentation offloads, adaptive interrupt moderation, and kernel-bypass frameworks like the Data Plane Development Kit and Express Network Adapter Express, the driver eliminates single-core CPU bottlenecks and minimizes memory copy overhead across high-bandwidth cloud instances.

Hardware Architecture and AWS Nitro Offload

Traditional cloud virtualization relies on a software hypervisor to emulate network hardware, intercepting every packet and introducing context switches between the guest operating system and the host. Amazon EC2 eliminates this bottleneck through the AWS Nitro System, which offloads networking, storage, and security processing onto dedicated ASICs and custom Nitro Cards.

The Elastic Network Adapter (ENA) is the custom network interface designed specifically to interact with these Nitro Cards. When an EC2 instance launches, the Nitro hardware presents the ENA device to the guest operating system over the PCIe bus using Single Root I/O Virtualization (SR-IOV). This allows the guest OS kernel to communicate directly with the physical network adapter's virtual function via Direct Memory Access (DMA), bypassing the hypervisor layer entirely during data plane operations.

To maintain flexibility across generations of EC2 instance types, the ENA driver separates control plane operations from data plane packet movement:

  • Admin Queue: A dedicated, low-bandwidth queue pair (Submission Queue and Completion Queue) used exclusively for device initialization, capability negotiation, queue creation, and statistics gathering. Because device features are negotiated dynamically over the Admin Queue at boot time, a single forward-compatible ENA driver can run across current and future EC2 instance generations without requiring kernel updates for new hardware revisions.
  • Async Event Notification Queue (AENQ): A specialized queue that allows the Nitro hardware to push asynchronous notifications—such as link state changes, configuration updates, or keep-alive events—directly to the host driver without polling.
  • I/O Queues: High-speed transmit (Tx) and receive (Rx) ring buffers designed solely for moving network packets with minimal CPU cycles.

Multi-Queue Scaling and Descriptor Ring Design

A single CPU core cannot process hundreds of gigabits per second of packet traffic when network speeds scale into tens of millions of packets per second. If packet throughput \(T\) in bits per second for a given packet size \(S\) in bytes and packet rate \(R\) in packets per second is defined as:

\[T = R \times S \times 8\]

Then achieving \(T = 200 \times 10^9\text{ bps}\) with standard \(1500\text{-byte}\) Ethernet frames requires processing approximately \(16.67 \times 10^6\text{ packets per second}\). At that rate, a single CPU core has less than \(60\text{ nanoseconds}\) per packet to handle interrupts, allocate memory, traverse the networking stack, and deliver data to user space.

To overcome single-core limits, the ENA driver implements a scale-out multi-queue architecture:

Receive Side Scaling (RSS)

When packets arrive at the Nitro card, hardware-based Receive Side Scaling computes a Toeplitz hash over the packet's 4-tuple (source IP, destination IP, source port, and destination port). Using the lower bits of this hash, the hardware looks up a configurable indirection table to assign the packet to a specific Rx queue. This ensures that all packets belonging to the same TCP or UDP flow land on the same Rx ring and are processed by the same vCPU, preserving packet ordering while distributing thousands of concurrent flows evenly across all available cores.

Transmit Packet Steering (XPS)

On the transmit path, the Linux kernel uses Transmit Packet Steering to map each vCPU to a dedicated Tx queue. When an application running on a specific core sends data, the ENA driver places the packet descriptors directly onto the Tx ring associated with that core. Because no other vCPU contends for that specific Tx ring, the driver processes transmissions without cross-core spinlock contention or cache line bouncing.

Low-Latency Mode (LLQ)

In standard DMA operations, the driver writes a packet descriptor to host RAM, rings a PCIe doorbell register, and waits for the network card to fetch the descriptor over the PCIe bus before fetching the packet payload. ENA supports Low-Latency Queue (LLQ) mode, where the driver pushes the transmit descriptors and packet headers directly into on-device memory via Memory-Mapped I/O (MMIO). This removes an entire PCIe round-trip read transaction before packet transmission begins.

Stateless Hardware Offloads and Jumbo Frames

Even when packet processing is distributed across dozens of vCPUs, per-byte operations such as checksum calculation and packet segmentation can saturate memory bandwidth. The ENA driver delegates these repetitive calculations to the Nitro hardware:

  • Checksum Offload: Rather than having the host CPU iterate over every byte of payload in software to compute IPv4, TCP, or UDP checksums, the ENA driver sets metadata flags in the Tx descriptor instructing the Nitro ASIC to compute and insert the checksum inline as the packet exits the card. On the receive side, the hardware verifies the checksum and passes a validation bit in the Rx completion descriptor.
  • TCP Segmentation Offload (TSO): When transmitting a large buffer (for example, \(64\text{ KB}\)), the kernel passes a single large super-frame to the ENA driver instead of slicing it into forty-five separate \(1500\text{-byte}\) packets in software. The Nitro hardware segments the payload into MTU-sized frames, replicates and updates the TCP/IP headers for each segment, and transmits them over the wire.
  • Generic Receive Offload (GRO): In the Linux driver, incoming packets from the same flow are coalesced into a single large packet buffer during the NAPI polling cycle before being passed up the kernel TCP/IP stack, drastically reducing per-packet protocol traversal overhead.
  • Jumbo Frames: Within an Amazon VPC, ENA supports a Maximum Transmission Unit (MTU) of up to \(9001\text{ bytes}\) instead of the standard \(1500\text{ bytes}\). Increasing the payload size by a factor of \(6\times\) reduces the packet rate required to saturate a high-bandwidth link by the same factor, directly multiplying CPU efficiency.

Interrupt Moderation and NAPI Polling

If the network card triggered a hardware interrupt (IRQ) for every arriving packet at \(100\text{ Gbps}\), the host vCPUs would spend \(100\%\) of their time saving registers and switching contexts—a failure mode known as receive livelock. The ENA driver avoids this using two cooperating mechanisms:

Linux NAPI (New API)

When the first packet arrives on an idle Rx queue, the ENA hardware fires an MSI-X interrupt pinned to that queue's assigned vCPU. The ENA interrupt handler immediately masks further hardware interrupts for that queue and schedules a softirq poll loop. The kernel then polls the Rx ring in batches (typically processing up to \(64\) packets per poll cycle) until the ring is drained, re-enabling hardware interrupts only when traffic subsides.

Adaptive Interrupt Moderation (DIM)

The ENA driver integrates with the Linux Dynamic Interrupt Moderation (DIM) framework and hardware interrupt coalescing timers. When packet rates are low, the interrupt delay timer is kept near zero so single packets wake the CPU immediately for minimal latency. When throughput surges, the moderation algorithm dynamically increases the coalescing timer and packet threshold, trading microseconds of latency for massive reductions in interrupt overhead.

Kernel Bypass with DPDK and XDP

For workloads that require extreme packet rates or ultra-low latency—such as virtual appliances, high-frequency trading, or distributed storage engines—the standard operating system kernel network stack still introduces overhead from socket buffer (sk_buff) allocations and system calls. ENA supports two primary paths around the standard stack:

Data Plane Development Kit (DPDK)

AWS provides an official ENA Poll Mode Driver (net_ena) in the upstream DPDK project. When bound via the vfio-pci kernel module, a user-space application takes direct ownership of the ENA PCIe queues. Using constant polling in user space without interrupts, zero-copy memory pools backed by hugepages, and SIMD vectorized descriptor processing, DPDK applications can drive millions of packets per second per core directly from user space.

eXpress Data Path (XDP)

Within the standard Linux kernel driver, ENA supports native XDP and AF_XDP zero-copy sockets. By executing an eBPF program inside the ENA driver's Rx poll function before the kernel allocates an sk_buff structure, the driver can drop malicious packets, redirect traffic to another interface, or pass frames directly into user-space memory rings at line rate.

ENA Express and the Scalable Reliable Datagram Protocol

While multi-queue RSS distributes hundreds of flows across multiple cores and network paths, a single TCP flow is traditionally hashed to a single path through the underlying data center fabric and handled by a single vCPU. On modern EC2 instances offering \(100\text{ Gbps}\) to \(400\text{ Gbps}\) or more of aggregate bandwidth, a single TCP flow is typically capped at \(5\text{ Gbps}\) to \(25\text{ Gbps}\) due to Equal-Cost Multi-Path (ECMP) routing collisions and TCP head-of-line blocking on packet loss.

To break this single-flow ceiling and deliver massive throughput effortlessly to unmodified applications, AWS introduced ENA Express, built on the Scalable Reliable Datagram (SRD) protocol running natively on the Nitro Card:

  1. Transparent Protocol Offload: The guest OS and ENA driver continue to send standard TCP or UDP packets. When ENA Express is enabled, the Nitro card intercepts these packets and encapsulates them into SRD packets before sending them across the AWS data center network.
  2. Multi-Path Spraying: Instead of pinning a flow to a single network path, SRD sprays individual packets of a single flow across dozens of parallel physical network paths simultaneously, routing around micro-burst congestion in real time.
  3. Out-of-Order Recovery and Reordering in Hardware: Because spraying packets across multiple paths causes them to arrive out of order, standard TCP would misinterpret the out-of-order arrivals as packet loss and collapse its congestion window. The receiving Nitro card handles packet acknowledgments, sub-millisecond retransmissions, and strict in-order reassembly entirely in hardware before placing the packets into the guest's ENA Rx queue.

By combining SRD hardware transport with multi-queue DMA, stateless offloads, and lockless per-core ring management, the ENA driver increases single-flow bandwidth up to \(25\text{ Gbps}\) and scales aggregate throughput into the hundreds of gigabits per second across modern Amazon EC2 instances.