Understanding IOMMU Groups for Linux PCIe Passthrough

This article provides an overview of Input-Output Memory Management Unit (IOMMU) groups and their critical role in configuring PCIe passthrough within the Linux operating system. It details how IOMMU hardware enforces isolation boundaries between devices, why this isolation directly determines whether a device can be safely passed to a virtual machine (VM), and how system hardware topology dictates device grouping.

What Is an IOMMU Group?

An IOMMU acts as a memory management unit for Direct Memory Access (DMA)-capable I/O buses, translating device-visible virtual addresses to physical memory addresses. In Linux, the kernel organizes hardware into "IOMMU groups." An IOMMU group represents the smallest set of hardware devices that can be isolated from all other devices in the system.

Devices end up in the same group when the hardware cannot guarantee that their transactions can be separated safely. If two PCIe devices share a bus or lack proper Access Control Services (ACS) support on the PCIe root port or switch, they can theoretically snoop on or modify each other's traffic without the CPU or IOMMU intercepting it. Consequently, the Linux kernel combines them into a single IOMMU group to reflect this shared trust boundary.

The Significance for PCIe Passthrough

When performing PCIe passthrough (commonly via VFIO in KVM/QEMU), the hypervisor grants a guest operating system direct, low-level control over physical hardware, such as a GPU or NVMe drive. IOMMU groups are significant for three key reasons:

All-or-Nothing Device Assignment The primary rule of VFIO passthrough is that isolation boundaries cannot be split. If you want to pass a GPU to a virtual machine, every other device inside that GPU’s IOMMU group must either be passed to the same VM or bound to a stub driver (like vfio-pci) to prevent host-level access. You cannot leave one device in the group under the host kernel’s control while passing another to a guest. Doing so creates a security vulnerability where the guest could potentially execute malicious DMA requests through the shared bus to compromise host memory.
Hardware and Data Integrity By ensuring that all devices within a single group are assigned together, the IOMMU protects host memory from errant or malicious DMA operations. This protects against host system crashes, data corruption, and unauthorized access to hypervisor memory by a compromised guest.
Dependency on Motherboard and CPU Architecture IOMMU groupings are largely determined by the physical design of the motherboard, the CPU PCIe lane layout, and the chipset's support for PCIe Access Control Services (ACS). High-end desktop (HEDT) and server platforms typically feature extensive ACS support, resulting in clean, one-device-per-group mappings. Consumer-grade platforms often bundle multiple PCIe slots, onboard audio, USB controllers, and SATA controllers into the same IOMMU group, limiting passthrough flexibility.

Dealing with Suboptimal IOMMU Groups

If a target device shares an IOMMU group with essential host components (such as the primary system storage or network interface), passthrough becomes impossible without structural changes. Users typically resolve this by:

Moving the PCIe Card: Relocating the target card to a PCIe slot wired directly to the CPU's primary lanes rather than slots routed through the motherboard chipset.
Toggling BIOS Settings: Enabling features like VT-d (Intel) or AMD-Vi, along with SR-IOV and ACS, to allow the kernel to delineate boundaries properly.
ACS Override Patch: A kernel-level modification that artificially splits IOMMU groups by ignoring missing hardware ACS capabilities. While this enables passthrough for problematic hardware topologies, it bypasses the physical isolation guarantees of the IOMMU and carries inherent stability and security risks.