How Linux Manages Nested Virtualization

Nested virtualization in Linux allows a virtual machine (VM) to act as a hypervisor and run its own virtual machines. Managed primarily through the Kernel-based Virtual Machine (KVM) module, the Linux operating system achieves this by abstracting, multiplexing, and translating hardware-assisted virtualization extensions across multiple software layers. This article explains the hierarchy of nested virtualization, how the Linux kernel handles CPU instruction trapping, the emulation of hardware control blocks, and multi-layered memory translation.

The Virtualization Hierarchy: L0, L1, and L2

To understand how Linux coordinates nested virtualization, it is necessary to define the three layers of execution:

• L0 (Host): The physical hardware running the bare-metal Linux operating system with the KVM module loaded.
• L1 (Guest Hypervisor): A virtual machine running on L0 that also runs its own Linux/KVM instance (or another hypervisor).
• L2 (Nested Guest): The virtual machine running inside the L1 guest hypervisor.

Physical CPU extensions—Intel VT-x (VMX) or AMD-V (SVM)—are designed natively to manage only a single hypervisor layer. Linux bridges this architectural gap by using L0 to intercept and emulate virtualization instructions issued by L1.

CPU Control Structures and Shadowing

Hardware virtualization relies on memory structures to store VM state and execution controls: the Virtual Machine Control Structure (VMCS) on Intel, or the Virtual Machine Control Block (VMCB) on AMD.

1. Software Emulation: L1 believes it has direct access to the CPU's virtualization features and creates its own VMCS/VMCB for L2. Because L1 does not have direct access to physical CPU registers, L0 intercepts (traps) L1's attempts to read or write to this structure and emulates the operations in software.
2. Hardware VMCS Shadowing: Emulating every control structure access creates significant performance overhead due to frequent CPU context switches (VM-exits). Modern Intel CPUs feature "VMCS Shadowing." Linux leverages this feature to point L1 to a hardware-managed "shadow VMCS." L1 can then execute read and write operations (VMREAD/VMWRITE) directly on the shadow structure without triggering an exit to L0. L0 only intervenes when L1 executes the launch command (VMLAUNCH/VMRESUME) to run L2.

Trapping and Forwarding VM-Exits

When L2 is running, any privileged instruction or hardware event causes a VM-exit. Because the physical CPU only knows about L0, control drops directly to the L0 Linux kernel:

1. Exit Analysis: L0 inspects the exit reason.
2. Direct Handling: If the exit was caused by a host-level interrupt (such as physical network traffic or host timer interrupts), L0 handles the interrupt and immediately resumes L2 execution without involving L1.
3. Reflective Exits (Forwarding): If the exit was caused by an event L1 configured to intercept (such as an L2 I/O operation or page fault), L0 emulates a hardware VM-exit inside L1. L0 updates L1's virtual CPU state to reflect the exit condition, switches the execution context to L1, and lets L1's hypervisor logic handle the event.

Memory Virtualization and Nested Page Tables

Memory management in nested virtualization requires mapping memory addresses across three layers:

• L2 Guest Physical Address (L2 GPA)
• L1 Guest Physical Address (L1 GPA)
• Host Physical Address (HPA)

CPUs use Extended Page Tables (EPT on Intel) or Nested Page Tables (NPT on AMD) for hardware-assisted address translation. Under nested conditions, translating an L2 address would normally require a computationally expensive two-dimensional page walk (translating L2 GPA to L1 GPA, then L1 GPA to HPA).

To optimize this, the Linux KVM module constructs a consolidated page table, often referred to as shadow EPT/NPT. KVM merges the L1-to-L0 mappings and the L2-to-L1 mappings into a single set of hardware page tables. The physical CPU uses this merged table to translate L2 memory access directly to physical host memory (HPA) in a single hardware-accelerated pass.