OS Kernel Optimizations for Fast UDP Processing

High-throughput UDP packet processing often encounters bottlenecks within the operating system kernel due to context switching, memory allocation overhead, interrupt handling, and lock contention. This article outlines key OS kernel-level optimizations—including socket buffer tuning, interrupt affinity, packet batching, network offloads, and modern in-kernel processing frameworks like XDP—to minimize packet drops, reduce latency, and maximize UDP throughput.

1. Tune Core Network Memory and Buffer Sizes

The Linux kernel relies on default buffer allocations that are often too small for high-bandwidth UDP traffic, leading to queue overflows and dropped packets.

• Increase Maximum Socket Receive/Send Buffers: Adjust global limits using sysctl to allow sockets to request larger buffers.
  • net.core.rmem_max = 67108864 (64 MB)
  • net.core.wmem_max = 67108864 (64 MB)
  • net.core.rmem_default = 33554432 (32 MB)
  • net.core.wmem_default = 33554432 (32 MB)
• Increase Network Device Backlog: Prevent the kernel from dropping packets before they reach the socket layer by expanding the input queue.
  • net.core.netdev_max_backlog = 10000
• Increase Socket-Level Buffers: Explicitly set SO_RCVBUF and SO_SNDBUF via setsockopt() in application code to match kernel limits.

2. Leverage UDP Offloads (GRO and GSO)

Generic Receive Offload (GRO) and Generic Segmentation Offload (GSO) reduce CPU utilization by combining multiple incoming packets into a single large structure (sk_buff) or splitting large outgoing buffers at the driver level.

• Enable UDP GRO: Allows the network stack to aggregate multiple UDP packets into a single buffer traversal, dramatically cutting per-packet processing costs.
• Enable UDP GSO: Enables the kernel to pass large multi-packet payloads directly to the network interface card (NIC), minimizing per-packet stack traversal on transmit.

3. Implement Multi-Queue and CPU Steering

Distributing the packet-processing workload evenly across multiple CPU cores prevents individual core saturation.

• RSS (Receive Side Scaling): Hardware-level hashing that directs incoming packets across multiple NIC hardware queues.
• RPS (Receive Packet Steering): A software implementation of RSS that distributes packet processing across CPUs at the driver level when hardware queues are limited.
• RFS (Receive Flow Steering): Routes packets to the CPU core where the target application thread is actively executing, improving CPU cache locality.
• IRQ Affinity: Pin NIC interrupt lines (IRQs) to dedicated CPU cores using smp_affinity to avoid expensive inter-core cache invalidations.

4. Reduce Syscall Overhead with Batching

Standard recv() and send() operations incur kernel-to-user space context switch overhead for every single packet.

• recvmmsg() and sendmmsg(): Replace single-packet system calls with batched calls to transmit or receive multiple messages in a single context switch.
• MSG_TRUNC and SO_BUSY_POLL: Enable low-latency socket polling (sysctl -w net.core.busy_poll=50 or per-socket SO_BUSY_POLL) to allow the driver to poll for new packets directly, bypassing standard interrupt overhead.

5. In-Kernel Processing with eBPF and XDP

When standard kernel network stack processing creates too much overhead, eXpress Data Path (XDP) provides a programmable, high-performance packet processing path.

• XDP Driver Mode: Executes eBPF bytecode directly at the network driver layer before the kernel allocates an sk_buff structure. This allows filtering, routing, or dropping UDP packets at line rate.
• AF_XDP (XSK): A high-speed socket type that redirects raw network frames directly to user space via a lockless ring buffer (zero-copy), bypassing the standard kernel TCP/IP stack entirely while retaining OS security boundaries.