Linux SYN Flood Mitigation Using TCP SYN Cookies

This article explains how the Linux operating system uses TCP SYN cookies to defend against SYN flood denial-of-service attacks. In a standard Transmission Control Protocol (TCP) handshake, incoming connection requests consume server memory while waiting for the client's final acknowledgment. When malicious actors exploit this design to deplete system resources, Linux dynamically deploys SYN cookies to offload connection state storage to the client, allowing legitimate connections to proceed without exhausting kernel memory.

The SYN Flood Vulnerability

Under standard TCP operation, establishing a connection requires a three-way handshake:

1. The client sends a SYN (synchronize) packet to the server.
2. The server responds with a SYN-ACK (synchronize-acknowledge) packet and allocates resources in a memory table known as the SYN backlog queue to track the half-open connection.
3. The client responds with an ACK packet, completing the handshake and moving the connection to the fully established state.

In a SYN flood attack, an attacker sends an overwhelming volume of SYN packets, often using spoofed IP addresses, and intentionally ignores the resulting SYN-ACK packets. The server’s SYN backlog queue fills up waiting for final ACK packets that never arrive. Once this queue reaches capacity (defined by net.ipv4.tcp_max_syn_backlog), the operating system drops subsequent incoming connection requests, denying service to legitimate users.

How TCP SYN Cookies Work

TCP SYN cookies eliminate the need to allocate memory during the initial SYN phase. Instead of allocating a transmission control block (TCB) in the SYN backlog, the Linux kernel encodes the essential connection state directly inside the 32-bit Initial Sequence Number (ISN) sent in the SYN-ACK packet. This sequence number acts as the "cookie."

When the SYN backlog queue becomes full, Linux generates this synthetic ISN using a specific bit breakdown:

• Bits 0–23 (24 bits): A cryptographic hash generated from the source IP address, source port, destination IP address, destination port, a slowly incrementing timestamp, and a secret server-side key.
• Bits 24–26 (3 bits): An encoded index representing the client's Maximum Segment Size (MSS) from a small, predefined table of values.
• Bits 27–31 (5 bits): A truncated timestamp counter, which increments every 64 seconds to validate that the response arrives within a valid time window.

Because the connection metadata is stored inside the sequence number itself, the server can safely discard the half-open connection from memory immediately after transmitting the SYN-ACK.

Handshake Completion and Verification

If the connection request is legitimate, the client replies with an ACK packet. In accordance with TCP specifications, the client sets the acknowledgment number to the server's sequence number plus one (\(\text{ISN} + 1\)).

Upon receiving this ACK, the Linux kernel processes the packet without needing an existing entry in the SYN backlog:

1. Extraction: The kernel subtracts 1 from the acknowledgment number to recover the original SYN cookie.
2. Timestamp Check: It inspects the timestamp bits to ensure the cookie has not expired (typically within a few minutes).
3. Cryptographic Validation: The kernel recomputes the cryptographic hash using the connection's four-tuple (IPs and ports) and the secret key. If the recomputed hash matches the hash stored in the cookie, the kernel verifies the client received the SYN-ACK.
4. Connection Allocation: The kernel extracts the MSS value, allocates the socket buffer and connection structures in memory, and transitions the connection directly to the ESTABLISHED state.

If the incoming packet is part of a flood attack, the attacker never sends the matching ACK, and the server consumes no persistent memory.

Configuring SYN Cookies in Linux

Linux controls SYN cookie behavior via the sysctl interface using the net.ipv4.tcp_syncookies parameter:

• 0 (Disabled): The kernel rejects new connections once the SYN backlog is exhausted.
• 1 (Enabled when needed): The default setting. SYN cookies activate automatically only when the SYN backlog queue overflows.
• 2 (Always enabled): SYN cookies are used unconditionally for all incoming connections, regardless of backlog saturation.

To check the current status on a Linux system, run:

sysctl net.ipv4.tcp_syncookies

To enable SYN cookies immediately and make the setting persistent across reboots, add or update the following line in /etc/sysctl.conf:

net.ipv4.tcp_syncookies = 1

Then apply the change:

sysctl -p

Trade-offs and Modern Adaptations

While SYN cookies effectively neutralize SYN floods, traditional implementations suffered from a major drawback: standard TCP options (such as window scaling, selective acknowledgments [SACK], and larger timestamps) could not fit into the 32-bit ISN and were consequently discarded.

Modern Linux kernels overcome this limitation by using TCP Timestamps (RFC 1323). When timestamps are enabled via net.ipv4.tcp_timestamps = 1, the Linux kernel encodes the client's supported TCP options into the lower bits of the 32-bit timestamp echo field, preserving advanced protocol features even while actively mitigating an attack.