How Linux Handles Symmetric Multiprocessing

This article explores how the Linux operating system manages symmetric multiprocessing (SMP), an architecture where multiple processor cores share a unified physical memory and system bus. Linux treats all available CPU cores symmetrically, leveraging advanced process scheduling, fine-grained concurrency controls, inter-processor communication, and memory topology awareness to distribute computational workloads efficiently while preventing resource conflicts.

The Evolution Away from the Giant Kernel Lock

In early versions of Linux SMP support, the kernel relied heavily on the Big Kernel Lock (BKL). This coarse-grained locking mechanism allowed only one processor core to execute kernel-space code at any given time. While simple, it caused significant bottlenecks as CPU core counts increased. Modern Linux kernels have entirely eliminated the BKL, replacing it with fine-grained synchronization primitives such as spinlocks, read-write locks, mutexes, and Read-Copy-Update (RCU). These primitives allow multiple cores to execute kernel routines simultaneously, provided they are not modifying the exact same kernel data structures.

Distributed Scheduling and Load Balancing

Linux handles workload distribution primarily through the scheduler—predominantly the Completely Fair Scheduler (CFS) and the newer Earliest Eligible Virtual Deadline First (EEVDF) scheduler. Under SMP:

• Per-CPU Runqueues: Instead of a single, centralized queue of tasks (which would create severe lock contention), each CPU core maintains its own independent runqueue.
• Scheduling Domains: The kernel groups logical CPUs into hierarchical scheduling domains based on physical topology (such as SMT threads, multi-core packages, and multi-socket nodes).
• Periodic Load Balancing: When a core's runqueue becomes unbalanced compared to others, or when a core becomes idle, the scheduler migrates tasks across cores. Load balancing evaluates the scheduling domain hierarchy to balance the system while minimizing the performance penalties of cross-core task migration.

Cache Affinity and NUMA Awareness

Migrating a process to a different CPU invalidates the task's cache data stored in L1 and L2 caches, resulting in memory latency penalties. To mitigate this, Linux enforces soft CPU affinity:

• Processor Affinity: The scheduler attempts to keep a task executing on the same CPU core to preserve cache warmth. Administrators can also set hard affinity rules using tools like taskset or control groups (cgroups).
• NUMA Support: In large-scale SMP systems, memory access times differ depending on which memory bank is closest to a given processor (Non-Uniform Memory Access). Linux SMP architecture incorporates NUMA awareness, allocating memory pages from the local memory node associated with the core executing the thread, thereby reducing cross-bus interconnect traffic.

Inter-Processor Interrupts (IPIs)

Independent cores must communicate to maintain system-wide state coherence. Linux utilizes Inter-Processor Interrupts (IPIs) at the hardware level for several critical coordination tasks:

• TLB Shootdowns: When a core modifies virtual memory page tables, it sends an IPI to other cores executing threads in that same address space to invalidate their Translation Lookaside Buffer (TLB) caches.
• Rescheduling: If a high-priority task is queued on a core currently running a lower-priority task, another core can signal it via an IPI to force an immediate context switch.
• System Halts and Profiling: Global operations, such as kernel debugging, profiling (e.g., perf), and system shutdown, rely on IPIs to synchronize or halt all active processing units simultaneously.