Understanding Linux blk-mq Storage I/O Framework

The blk-mq (Block Multi-Queue) framework is a modern architecture within the Linux kernel designed to parallelize block I/O processing across multi-core processors and high-speed solid-state storage. By replacing the legacy single-queue I/O model with a two-tiered queuing design, blk-mq eliminates lock contention, significantly decreases I/O latency, and scales input/output operations per second (IOPS) to match the capabilities of modern devices like NVMe SSDs.

The Problem with Legacy Block I/O

Historically, the Linux block layer relied on a single request queue protected by a single lock. This architecture was sufficient for rotational hard disk drives (HDDs), where hardware latencies were measured in milliseconds and I/O processing was bound by mechanical drive heads rather than CPU speed.

As multi-core CPUs became standard and solid-state storage (such as PCIe and NVMe SSDs) emerged, storage latencies dropped to microseconds while throughput surged to millions of IOPS. Under these workloads, the legacy architecture became a critical bottleneck. Multiple CPU cores simultaneously competing for a single lock to submit I/O requests caused severe lock contention, resulting in high CPU overhead and underutilized storage hardware.

Architecture of blk-mq

Introduced in Linux kernel 3.13 and made the mandatory standard in Linux 5.0, blk-mq resolves this bottleneck by splitting the queuing mechanism into two distinct, scalable layers:

1. Software Staging Queues (Per-CPU Queues): When an application submits an I/O request, the request is initially placed into a software staging queue. These queues are allocated per-CPU (or per-core), meaning a thread running on a specific core can push an I/O request into its local queue without needing to acquire locks across other CPU cores. This local queuing virtually eliminates cross-CPU synchronization overhead.

2. Hardware Dispatch Queues: Modern storage controllers, particularly NVMe and high-end SAS/SCSI controllers, inherently support multiple hardware submission queues. The blk-mq framework allocates hardware dispatch queues that map directly to the queues provided by the storage controller.

3. Queue Mapping: The framework maps the software queues onto the available hardware queues. If the hardware supports as many queues as there are CPU cores, there is a 1:1 mapping with no lock contention at all. If the hardware has fewer queues than CPU cores, multiple software queues share a hardware queue using lightweight, fine-grained locking, which is still dramatically faster than a single global lock.

Key Functions and Benefits

• Scalable Tag Allocation: blk-mq uses a specialized lockless bitmap structure (sbitmap) to allocate command identifiers (tags) for tracking in-flight I/O requests, avoiding contention during request tracking.
• NUMA Optimization: The architecture is Non-Uniform Memory Access (NUMA) aware, routing requests through queues and memory nodes that are physically closest to the core handling the task.
• Full Hardware Utilization: Fast storage devices are able to process hundreds of thousands or millions of IOPS concurrently without being throttled by the operating system kernel.
• Support for Legacy Hardware: While optimized for NVMe, blk-mq also services traditional SCSI and SATA devices via compatibility mapping layers, ensuring even slower devices benefit from lower CPU overhead during request processing.