Linux IPsec Implementation Using StrongSwan

This article explores how the Linux operating system implements IPsec via the StrongSwan software suite. It outlines the architectural separation between user-space key exchange and kernel-space packet processing, details the role of the Linux XFRM framework, explains how the Netlink interface facilitates communication between StrongSwan and the kernel, and breaks down the end-to-end packet processing lifecycle for secure network communications.

Architectural Separation: Control Plane vs. Data Plane

Linux implements IPsec by dividing responsibilities between the operating system kernel and user-space applications:

• Data Plane (Kernel Space): The Linux kernel natively handles packet interception, encryption, decryption, authentication, and encapsulation. It does not handle key negotiations or authentication handshakes directly.
• Control Plane (User Space): StrongSwan operates primarily in user space. Its core daemon, charon, manages the Internet Key Exchange protocols (IKEv1 and IKEv2), authenticates endpoints using certificates or pre-shared keys (PSKs), and negotiates cryptographic parameters.

Once StrongSwan completes the control-plane negotiations, it offloads the actual cryptographic keys and security policies to the Linux kernel.

The Linux XFRM Framework

The Linux kernel manages IPsec through an internal architecture called XFRM (pronounced "transform"). XFRM maintains two core in-memory databases:

1. Security Policy Database (SPD): Defines what traffic must be protected. Policies specify matching criteria (source/destination IP addresses, ports, and protocols) and the required action (e.g., encrypt, bypass, or discard).
2. Security Association Database (SAD): Defines how matching traffic is protected. Each Security Association (SA) stores the unidirectional cryptographic parameters, including encryption/integrity keys, algorithms, cipher modes, and the Security Parameter Index (SPI).

StrongSwan and Kernel Communication via Netlink

StrongSwan communicates with the kernel's XFRM framework using the Netlink socket interface (specifically the NETLINK_XFRM family).

1. Policy and State Injection: Upon successful IKE negotiation, StrongSwan issues Netlink messages to instantiate SAs and SPs inside the kernel.
2. Kernel Events: If the kernel detects outbound traffic matching an SP that lacks an active SA, it emits an XFRM_MSG_ACQUIRE message via Netlink. StrongSwan listens for these acquire triggers to initiate on-demand IKE negotiations.
3. Lifetime Expirations: The kernel monitors byte and time limits for active SAs. When an SA nears expiration, it notifies StrongSwan via Netlink, allowing the daemon to rekey the connection without dropping packets.

Packet Processing Workflow

Outbound Traffic (Transmission)

1. Policy Lookup: An application generates a packet. The kernel routing layer checks the XFRM SPD.
2. SA Resolution: If an SP requires IPsec, the kernel queries the SAD using the policy's parameters.
3. Encapsulation and Encryption: The kernel applies the specified transformation (such as Encapsulating Security Payload, or ESP) using cryptographic drivers via the Linux Crypto API.
4. Transmission: The encrypted packet, encapsulated with a new outer IP header containing the SPI, is routed out through the physical network interface.

Inbound Traffic (Reception)

1. Packet Interception: An incoming ESP packet arrives at the network interface.
2. SA Lookup: The kernel extracts the SPI and destination IP from the packet header and queries the SAD.
3. Decryption and Verification: The kernel decrypts the payload and verifies integrity hashes.
4. Policy Verification: The kernel validates that the decrypted packet matches an authorized policy in the SPD before passing the plaintext payload to the local network stack.

Implementation Modes in Modern Linux

StrongSwan and Linux support two primary architectural models for handling IPsec traffic:

• Policy-Based IPsec: The classic implementation where traffic is routed implicitly based on SPD matching rules without dedicated virtual interfaces.
• Route-Based IPsec (XFRM Interfaces): Modern Linux kernels (v4.19+) allow StrongSwan to bind IPsec policies to explicit virtual network interfaces (xfrm interfaces). This approach simplifies configuration by integrating IPsec directly with standard Linux routing tables, firewall rules (iptables/nftables), and traffic control subsystems.