How can you use ENA Express to reduce EC2 tail latency?

Amazon Elastic Network Adapter Express is a networking feature powered by the AWS Nitro System and the Scalable Reliable Datagram protocol that significantly lowers tail latency and increases single-flow bandwidth between Amazon EC2 instances. By dynamically routing network packets across multiple paths and handling packet recovery directly on the Nitro card hardware, ENA Express overcomes traditional TCP congestion bottlenecks. This article explains how ENA Express works under the hood, its prerequisites, and how to enable, configure, and monitor it on your EC2 instances to improve performance for latency-sensitive workloads.

Understanding Tail Latency in Cloud Networking

In distributed applications such as financial trading platforms, real-time analytics, and in-memory databases, average network latency only tells part of the performance story. Tail latency, commonly measured at the 99th (\(P_{99}\)) or 99.9th (\(P_{99.9}\)) percentile, represents the slowest network responses in a system. Even if 99% of your requests complete in microseconds, the remaining 1% experiencing millisecond delays can stall distributed transactions and degrade overall application throughput.

Traditional TCP/IP networking relies on Equal-Cost Multi-Path (ECMP) routing, which hashes a network flow's 5-tuple (source IP, destination IP, source port, destination port, and protocol) to assign all packets of that flow to a single physical network path. While this prevents packets from arriving out of order, it creates two major challenges:

  1. Hash collisions: Multiple high-bandwidth flows can be hashed onto the same network link, causing congestion and buffer buildup even when adjacent paths are idle.
  2. Slow failure recovery: If a network switch or link degrades, TCP must wait for retransmission timeouts to detect packet loss and scale back its congestion window, resulting in significant latency spikes.

How ENA Express and SRD Work

ENA Express addresses the limitations of traditional ECMP routing by replacing standard physical transport mechanisms with the AWS Scalable Reliable Datagram (SRD) protocol directly on the AWS Nitro hardware. Importantly, ENA Express is completely transparent to your applications; it works with standard TCP and UDP sockets without requiring application code changes.

When an EC2 instance sends a TCP or UDP packet over an ENA Express-enabled interface, the Nitro card encapsulates the traffic into SRD packets and manages delivery across the AWS data center fabric using four core mechanisms:

  • Multi-path spraying: Instead of pinning a single flow to a single network path, SRD sprays packets from a single flow across dozens of available network paths simultaneously. This balances load evenly across the network fabric and prevents single-link congestion.
  • Fine-grained congestion control: SRD continuously monitors round-trip times and link utilization at the hardware level, dynamically routing packets around congested or failed links in microseconds rather than waiting for operating system TCP timeouts.
  • Hardware retransmission: If a packet is lost in transit, the Nitro card detects the missing sequence and retransmits the packet directly between the source and destination Nitro cards, hiding the loss from the guest operating system's TCP stack.
  • Out-of-order reassembly: Because packets take different paths, they naturally arrive out of order. For TCP traffic, the receiving Nitro card buffers and reorders the packets in hardware before passing them to the guest OS, preventing the OS from mistakenly interpreting out-of-order delivery as packet loss.

By offloading path selection, retransmission, and reordering to the Nitro card, ENA Express achieves two primary performance improvements:

\[\text{Single-Flow Bandwidth}: 5\text{ Gbps} \rightarrow 25\text{ Gbps}\]

\[\text{Tail Latency } (P_{99.9}): \text{Up to } 85\% \text{ reduction under high load}\]

Prerequisites and Compatibility

Before enabling ENA Express, both the source and destination EC2 instances must meet specific architectural requirements:

  • Supported instance types: Both instances must support ENA Express. This generally includes modern Nitro-based instances with high networking capabilities, such as c6gn.16xlarge, c6in.32xlarge, c7g.16xlarge, c7gn.16xlarge, r6in.32xlarge, m6in.32xlarge, hpc7g, and newer high-bandwidth instance families.
  • Same Availability Zone: The communicating instances must reside in the exact same Availability Zone (AZ). Traffic crossing AZ boundaries or going through transit gateways falls back to standard ENA routing.
  • ENA Linux driver version: For Linux instances, the ena kernel driver must be version 2.8.0 or later to view ENA Express metrics and manage settings via ethtool.
  • Maximum Transmission Unit (MTU): To achieve optimal throughput and avoid fragmentation overhead, configure your VPC and instance network interfaces to use Jumbo Frames (MTU 8900 or 9001, depending on exact topology requirements; standard VPC Jumbo Frames use 9001, and ENA Express handles encapsulation internally).

You can verify whether a specific instance type supports ENA Express using the AWS CLI:

aws ec2 describe-instance-types \
  --instance-types c7gn.16xlarge \
  --query "InstanceTypes[].NetworkInfo.EnaExpressSupport" \
  --output text

How to Enable ENA Express

ENA Express is configured per Elastic Network Interface (ENI). You can enable it when launching a new instance or attach/modify it on an existing network interface using the AWS CLI, AWS Management Console, or Infrastructure as Code tools.

Enabling on an Existing Network Interface

To enable ENA Express on an existing ENI, use the modify-network-interface-attribute command:

aws ec2 modify-network-interface-attribute \
  --network-interface-id eni-0123456789abcdef0 \
  --ena-srd-specification EnaSrdEnabled=true

Enabling UDP Support for ENA Express

By default, enabling EnaSrdEnabled activates SRD transport for both TCP and UDP traffic. However, because SRD sprays packets across multiple paths, UDP packets may arrive out of order (unlike TCP, the Nitro card does not reorder UDP packets so as not to introduce head-of-line blocking for real-time protocols). If your UDP application cannot tolerate out-of-order packets, you can disable SRD specifically for UDP while keeping it enabled for TCP:

aws ec2 modify-network-interface-attribute \
  --network-interface-id eni-0123456789abcdef0 \
  --ena-srd-specification "EnaSrdEnabled=true,EnaSrdUdpSpecification={EnaSrdUdpEnabled=false}"

Enabling at Instance Launch

You can also enable ENA Express directly during instance launch using run-instances:

aws ec2 run-instances \
  --image-id ami-0abcdef1234567890 \
  --instance-type c7gn.16xlarge \
  --subnet-id subnet-0123456789abcdef0 \
  --network-interfaces "DeviceIndex=0,EnaSrdSpecification={EnaSrdEnabled=true,EnaSrdUdpSpecification={EnaSrdUdpEnabled=true}}"

Guest OS Configuration and Tuning

Once ENA Express is enabled on the ENI, traffic between two compatible instances in the same AZ automatically uses SRD. However, to maximize single-flow throughput (up to 25 Gbps) and keep tail latency low, you should verify your guest operating system configuration.

1. Verify the ENA Driver Version

Check that your Linux kernel is running ENA driver version 2.8.0 or newer:

ethtool -i eth0

Look for the version: field in the output. If it is older than 2.8.0, update your Linux kernel or install the latest amzn-drivers package.

2. Tune TCP Buffer Sizes for 25 Gbps Flows

Standard Linux TCP buffer defaults are often sized for 5 Gbps or 10 Gbps flows. To allow a single TCP connection to scale up to the 25 Gbps limit provided by ENA Express, calculate the Bandwidth-Delay Product (BDP):

\[\text{BDP} = \text{Bandwidth} \times \text{RTT}\]

For a \(25\text{ Gbps}\) (\(3.125 \times 10^9\text{ bytes/sec}\)) flow with a conservative intra-AZ round-trip time of \(500\ \mu\text{s}\) (\(0.0005\text{ seconds}\)), the minimum buffer size required is approximately \(1.56\text{ MB}\). To account for transient queuing and window scaling, increase the maximum TCP read and write buffers in /etc/sysctl.conf:

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

Apply the settings without rebooting:

sudo sysctl -p

Monitoring and Verifying ENA Express

To confirm that your traffic is actively using SRD and to monitor hardware resource utilization, inspect the extended network statistics exposed by ethtool.

Run the following command on your instance (replace eth0 with your primary network interface name, such as ens5):

ethtool -S eth0 | grep ena_srd

This command outputs several key metrics:

  • ena_srd_mode: Displays a bitmask indicating the active configuration (1 means ENA Express is enabled for TCP; 3 means both TCP and UDP are enabled).
  • ena_srd_tx_pkts: The number of packets transmitted over SRD. If this counter is incrementing, your outbound traffic is actively using ENA Express.
  • ena_srd_rx_pkts: The number of packets received over SRD.
  • ena_srd_eligible_tx_pkts: The number of packets eligible to be sent over SRD. Comparing ena_srd_tx_pkts to ena_srd_eligible_tx_pkts gives you the SRD utilization ratio:

\[\text{SRD Utilization} = \frac{\text{ena\_srd\_tx\_pkts}}{\text{ena\_srd\_eligible\_tx\_pkts}} \times 100\%\]

  • ena_srd_resource_utilization: Shows the percentage of hardware SRD connection tracking resources currently in use on the Nitro card. If this value approaches 100%, new connections may temporarily fall back to standard ENA routing until existing flows close.

Best Practices for Reducing Tail Latency

  1. Combine with Cluster Placement Groups: While ENA Express prevents multi-path congestion across the AZ, combining it with a Cluster Placement Group places your instances in close physical proximity, minimizing both baseline propagation delay (\(P_{50}\)) and worst-case tail latency (\(P_{99.9}\)).
  2. Monitor ena_srd_resource_utilization: ENA Express excels at high-throughput, medium-connection-count workloads (such as storage replication, distributed caching, and media pipelines). If your workload opens hundreds of thousands of concurrent connections, monitor the resource utilization counter to ensure the Nitro hardware buffers are not exhausted.
  3. Keep TCP In-Order Guarantees in Mind for Custom Protocols: If you build custom protocols over UDP, test thoroughly before enabling EnaSrdUdpEnabled, or implement sequence handling in the application layer to accommodate multi-path packet reordering.