Linux VXLAN Encapsulation for Overlay Networks

This article provides an overview of how the Linux operating system handles Virtual Extensible LAN (VXLAN) encapsulation within its networking stack. It covers the role of the Linux kernel in virtualizing Layer 2 networks over Layer 3 infrastructures, detailing the packet transmission and reception pipelines, interface configuration via native tooling, and how the kernel tracks endpoints using the Forwarding Database (FDB).

Understanding VXLAN in the Linux Kernel

VXLAN is an overlay network protocol that encapsulates Layer 2 Ethernet frames into Layer 3 UDP packets, traditionally destined for UDP port 4789. This allows virtualized environments, such as containers and virtual machines, to communicate over isolated Layer 2 broadcast domains even when separated by Layer 3 routing boundaries.

The Linux kernel has supported native VXLAN functionality since version 3.7 through the vxlan kernel module. In Linux, a VXLAN Tunnel Endpoint (VTEP) is represented as a virtual network device. This device terminates VXLAN tunnels, handling both the encapsulation of outgoing packets and the decapsulation of incoming packets directly within the kernel's network subsystem.

The Encapsulation Process (Transmitting Data)

When a virtual workload (such as a container or VM connected to a Linux bridge) sends a frame to a remote host across the overlay, the Linux kernel processes the packet through the following stages:

1. Packet Ingress to the Bridge: The application generates a standard Layer 2 Ethernet frame. This frame enters a Linux bridge or an Open vSwitch (OVS) datapath to which the VXLAN interface is attached.
2. FDB Lookup: The kernel inspects the destination MAC address of the inner frame against the VXLAN device's internal Forwarding Database (FDB). The FDB determines the outer destination IP address—the remote VTEP—associated with that target MAC address.
3. Encapsulation: The kernel prepends headers to the original Layer 2 frame:
   • VXLAN Header: An 8-byte header that contains the 24-bit VXLAN Network Identifier (VNI), which uniquely identifies the overlay segment.
   • UDP Header: An 8-byte header specifying the source port (often generated via an inner packet hash to allow Equal-Cost Multi-Path [ECMP] hashing across the underlay network) and the destination port (usually 4789).
   • Outer IP Header: The source IP is the local host's physical network interface, and the destination IP is the remote VTEP identified during the FDB lookup.
   • Outer Ethernet Header: Formed based on the host routing table to reach the next-hop gateway on the physical underlay.
4. Transmission: The fully encapsulated packet is treated as standard UDP traffic and routed out of the physical network interface card (NIC).

The Decapsulation Process (Receiving Data)

When encapsulated traffic arrives from the underlay network, the kernel processes it in reverse:

1. UDP Reception: The physical NIC receives an IP/UDP packet. The kernel's UDP socket layer receives the packet and matches the destination port (4789) to the listening VXLAN driver.
2. Header Stripping: The VXLAN driver extracts the 24-bit VNI from the VXLAN header to identify the correct local VXLAN virtual interface. It then strips the outer Ethernet, outer IP, UDP, and VXLAN headers.
3. MAC Learning: If learning is enabled, the kernel inspects the inner source MAC address and maps it to the sender's outer IP address in the local FDB, updating its routing logic for future return traffic.
4. Delivery to Target: The original inner Ethernet frame is injected back into the local networking stack (e.g., passed to the attached bridge), which inspects the inner destination MAC and switches the frame to the destination container or virtual machine.

Forwarding Database (FDB) and Control Plane

The Linux kernel needs to know which remote VTEP IP corresponds to which inner destination MAC address. Linux supports two primary methods to manage this information:

Data-Plane Learning (Multicast / Flooding)

In simpler setups, Linux relies on traditional flood-and-learn mechanics:

• BUM Traffic Handling: Broadcast, Unknown unicast, and Multicast (BUM) traffic is replicated by sending it to an underlay multicast group or using an explicit list of remote VTEP unicast IPs (head-end replication).
• Dynamic FDB Population: When a reply arrives, the kernel observes the inner source MAC and outer source IP, dynamically creating an entry in the FDB with an aging timer.

Control-Plane Driven (Static / EVPN)

In production environments (such as Kubernetes with Cilium/Calico or data centers using FRRouting with BGP EVPN), data-plane learning is often disabled (nolearning). Instead:

• Network controllers or routing daemons directly inject static FDB entries into the kernel using Netlink sockets (ip neighbor commands).
• The Linux kernel does not perform packet flooding; if an entry is not present in the FDB, the packet can be forwarded to a default fallback VTEP or dropped, minimizing underlay broadcast traffic.

Offloading and Performance Considerations

Encapsulation introduces CPU overhead due to packet construction and increased header sizes. The Linux kernel mitigates this through:

• Hardware Offloading: Modern NICs support VXLAN Task Offload, allowing the physical hardware to perform checksum calculations (UDP/IP checksums) and Large Send Offload (LSO/TSO) on the encapsulated inner frames.
• Path MTU: Because the combined outer headers (Ethernet, IP, UDP, VXLAN) typically consume 50 bytes, the underlay network MTU must be increased (e.g., to 1600 or jumbo frames of 9000 bytes) to prevent packet fragmentation when using standard 1500-byte payloads in the overlay.