Which Virtualization Technologies Have Powered Amazon EC2?

Amazon Elastic Compute Cloud has transitioned through several major virtualization architectures since its initial beta launch in 2006. Starting with a heavily customized version of the Xen hypervisor utilizing paravirtualization, Amazon Web Services later adopted hardware-assisted virtualization to improve performance and support broader operating systems. In 2017, Amazon introduced the AWS Nitro System, which offloaded virtualization management, networking, and storage functions to dedicated hardware cards alongside a lightweight KVM-based hypervisor. This evolution enabled modern virtualized instances, bare-metal instances, and specialized micro-virtualization architectures that power cloud computing workloads today.

The Early Era: Xen Paravirtualization (PV)

When Amazon EC2 launched in August 2006, hardware-assisted virtualization extensions from CPU manufacturers (such as Intel VT-x and AMD-V) were either brand new, unavailable at scale, or carried significant performance penalties. To slice physical servers into isolated virtual machines efficiently, AWS built its initial infrastructure on the open-source Xen hypervisor using Paravirtualization (PV).

In a traditional x86 architecture, processor privilege levels are divided into four protection rings, numbered from \(0\) (most privileged) to \(3\) (least privileged):

\[\text{Ring } 0 \text{ (Kernel)} > \text{Ring } 1 > \text{Ring } 2 > \text{Ring } 3 \text{ (User Space)}\]

Under Xen PV, the Xen hypervisor itself ran in \(\text{Ring } 0\) to maintain total control over the physical hardware. Because the guest operating system kernel could no longer run in \(\text{Ring } 0\), its kernel was modified to run in \(\text{Ring } 1\) (or \(\text{Ring } 3\) on 64-bit x86 architectures), while user applications remained in \(\text{Ring } 3\).

Key characteristics of Xen Paravirtualization on EC2 included:

• Modified Guest Kernels: Guest operating systems had to be explicitly aware that they were running in a virtualized environment. Standard operating systems could not boot unmodified; instead, they relied on special kernel builds (originally managed via Amazon Kernel Images, or AKIs, and later loaded via PvGrub).
• Hypercalls: Instead of executing privileged CPU instructions that would trap and fail outside of \(\text{Ring } 0\), the modified guest kernel made direct synchronous software calls—known as hypercalls—directly to the Xen hypervisor to handle memory management, page table updates, and I/O operations.
• Domain 0 (Dom0) Bottlenecks: In the Xen architecture, a privileged control domain known as Dom0 ran alongside guest instances (DomU). Every network packet and storage request from a guest instance had to pass through Dom0 software emulation and routing layers before reaching physical hardware, which introduced jitter and consumed host CPU cycles.

Transition to Xen Hardware Virtual Machine (HVM)

As Intel and AMD matured their hardware virtualization extensions—specifically Intel VT-x with Extended Page Tables (EPT) and AMD-V with Rapid Virtualization Indexing (RVI)—AWS introduced Hardware Virtual Machine (HVM) instances.

Xen HVM allowed guest operating systems to run completely unmodified, as if they were executing on bare-metal hardware. Processors introduced a new privilege mode altogether: root mode for the hypervisor and non-root mode for the guest virtual machines, allowing the guest OS kernel to reside in its native \(\text{Ring } 0\) within non-root mode.

The transition from PV to HVM occurred in stages:

1. Pure HVM: Allowed unmodified operating systems such as Microsoft Windows Server to run on EC2, alongside standard Linux distributions using their native bootloaders. However, early HVM relied on emulated hardware for disk and network controllers, which was slower than paravirtualized I/O.
2. PV on HVM: To combine the CPU and memory benefits of hardware virtualization with the I/O speed of paravirtualization, AWS adopted "PV on HVM" drivers. Guest operating systems ran unmodified in hardware-assisted mode but loaded specialized Xen paravirtualized drivers for network and storage access, bypassing slow hardware emulation.
3. SR-IOV and Enhanced Networking: As instance sizes grew, passing network and storage traffic through Dom0 remained a latency bottleneck. AWS began integrating Single Root I/O Virtualization (SR-IOV), allowing guest instances to bypass Dom0 for data plane operations and communicate directly with physical network interface cards (NICs).

The Annapurna Labs Acquisition and the Road to Nitro

Despite optimizations to Xen HVM, the fundamental architecture still required the host CPU and system RAM to run Dom0 management software, software-defined networking (VPC encapsulation), and storage encryption. For a host server with \(N\) physical CPU cores, a noticeable fraction of compute and memory had to be reserved for the hypervisor and control plane rather than sold to customers:

\[C_{\text{available}} = C_{\text{total}} - C_{\text{Dom0}} - C_{\text{hypervisor}}\]

In 2013, AWS began offloading networking tasks to custom hardware, culminating in the January 2015 acquisition of Annapurna Labs, an Israeli microelectronics company. This acquisition laid the groundwork for a complete architectural overhaul of EC2 virtualization: moving the control plane, I/O virtualization, and security monitoring off the main server motherboard and onto dedicated Application-Specific Integrated Circuits (ASICs).

The AWS Nitro System and the KVM Hypervisor

Officially unveiled in November 2017 with the launch of the C5 instance family, the AWS Nitro System replaced the legacy Xen architecture for all new EC2 instance generations. Rather than relying on a heavy software layer running on the host processor, the Nitro System breaks virtualization down into three modular hardware and firmware components:

1. Nitro Cards

Dedicated PCIe hardware cards powered by custom Annapurna Labs ASICs handle all I/O functions. Separate Nitro Cards manage Amazon Virtual Private Cloud (VPC) networking, Amazon Elastic Block Store (EBS) volume attachment and encryption, local NVMe instance storage, and system control. To the main host CPU, these cards present themselves simply as standard PCIe devices (using NVMe and Elastic Network Adapter [ENA] interfaces) via SR-IOV.

2. Nitro Security Chip

Integrated directly onto the host motherboard, the Nitro Security Chip monitors and protects hardware resources. It establishes a hardware root of trust, verifies firmware integrity at boot, and blocks unauthorized access or modification to non-volatile memory from software running on the host CPU.

3. The Nitro Hypervisor (KVM-based)

Because networking, storage, and management tasks are handled entirely by the dedicated Nitro Cards, Dom0 is eliminated. In place of Xen, AWS implemented the Nitro Hypervisor, which is built on core Linux Kernel-based Virtual Machine (KVM) technology.

Unlike standard general-purpose KVM deployments, the Nitro Hypervisor is stripped down to a minimal firmware-like footprint. It does not include a general-purpose operating system, networking stack, or device drivers, and it never initiates network connections. Its sole responsibility is partitioning the host CPU and physical memory (\(\text{RAM}\)) among guest instances. As a result, nearly \(100\%\) of the host server's compute and memory resources are delivered directly to customer instances, with performance indistinguishable from bare metal.

Bare Metal Instances (*.metal)

A direct consequence of shifting virtualization management and I/O to Nitro Cards is the ability to offer EC2 Bare Metal instances (such as i3.metal, c5.metal, and m7i.metal).

Because the Nitro Cards and Nitro Security Chip control boot orchestration, EBS storage, and VPC networking independently of the host CPU, AWS can boot an instance without loading the Nitro Hypervisor at all. The customer's operating system boots directly on the physical processor and memory, retaining full access to hardware performance counters and Intel VT-x or AMD-V extensions. This enables customers to run their own hypervisors (such as VMware ESXi or KVM) or licensing-restricted workloads directly on EC2 hardware while still utilizing cloud-native EBS volumes and VPC networking.

Specialized Virtualization on Modern EC2

Beyond standard x86 and Arm (AWS Graviton) virtual machines, the Nitro foundation has enabled specialized virtualization models for specific workloads:

Mac Instances

Introduced in 2020, EC2 Mac instances run on physical Apple Mac mini hardware (featuring Intel or Apple Silicon M-series processors) connected to the AWS Nitro System via high-speed Thunderbolt interfaces. The Nitro Controller manages the physical Mac as a bare-metal host, providing it with EBS storage and VPC networking while running macOS natively or through Apple’s native Virtualization framework.

Nitro Enclaves

To support workloads processing highly sensitive data (such as cryptographic keys or personally identifiable information), AWS introduced Nitro Enclaves. Using the same Nitro Hypervisor technology that isolates EC2 instances from one another, a customer can carve out CPU cores and memory from an existing EC2 instance to create an isolated, hardened virtual machine. An enclave has no persistent storage, no interactive shell access, and no external networking—communicating with its parent instance solely over a local virtual socket (vsock).

Firecracker MicroVMs

While not used directly to boot standard customer EC2 instances, AWS developed Firecracker in 2018 as an open-source Virtual Machine Monitor (VMM) written in Rust that utilizes KVM. Running atop EC2 .metal instances, Firecracker creates lightweight microVMs in milliseconds with a memory overhead of less than \(5\text{ MiB}\) per microVM. This specialized layer powers AWS serverless compute services, including AWS Lambda and AWS Fargate, combining the multi-tenant hardware isolation of KVM virtualization with the rapid startup speed of containers.

Summary of EC2 Virtualization Generations