How Do Trusted Execution Environments Encrypt Memory?

Trusted Execution Environments are hardware-isolated regions of a processor that protect application code and data inside encrypted memory enclaves, even when the operating system kernel or hypervisor is compromised. By shifting the root of trust from privileged software to the CPU silicon itself, these environments ensure that unauthorized software, including the kernel, cannot read or tamper with sensitive data in RAM.

Shifting the Root of Trust to Hardware

In traditional computing, the operating system kernel and hypervisor sit at the highest privilege levels. Because they manage page tables, memory allocation, and hardware interrupts, a compromised kernel can normally inspect the memory of any user-space application.

Trusted Execution Environments (TEEs), such as Intel SGX, AMD SEV, and ARM Confidential Compute Architecture, invert this trust model. They shrink the Trusted Computing Base (TCB) to include only the CPU package and the enclave's own code. The kernel is treated as an untrusted resource manager: it can still schedule threads and allocate physical memory pages, but the CPU hardware blocks it from reading the plaintext contents of those pages.

Hardware Memory Encryption Engines

The core mechanism that protects enclave data in main memory (DRAM) is a dedicated hardware memory encryption engine located inside the processor's memory controller. Whenever data leaves the CPU's on-die caches to be written to DRAM, the hardware automatically encrypts it. When the enclave reads that data back into the CPU caches, the hardware decrypts it on the fly.

Most modern TEEs use a tweakable block cipher mode such as AES-XTS or AES-GCM. In a tweakable cipher, the physical memory address \(A\) acts as a tweak alongside a hardware-protected ephemeral key \(K\) generated at boot time:

\[C = E_K(P, A)\]

Here, \(P\) is the plaintext cache line, \(C\) is the ciphertext stored in DRAM, and \(A\) is the physical address. Because the key \(K\) never leaves the CPU silicon and is inaccessible to any software instruction, physical attacks like cold-boot attacks or bus snooping only yield encrypted data.

Access Control and Page Table Isolation

Encryption alone does not stop a compromised kernel from remapping memory pages or attempting to read enclave addresses. To prevent this, the CPU enforces strict hardware access controls on top of standard virtual-to-physical address translation.

During enclave initialization, the CPU marks specific physical memory regions as protected, often tracking them in a hardware-managed structure such as an Enclave Page Cache Map (EPCM) or Reverse Map Table (RMP). Before any memory access completes, the processor's Memory Management Unit (MMU) checks this structure to verify:

  • The identity of the requester matches the enclave that owns the page.
  • The physical page is mapped to the expected virtual address.
  • No untrusted software, including the kernel or DMA-capable peripherals, is attempting to read or write the protected region.

If the kernel tries to access an enclave page, the CPU either triggers a hardware fault or returns meaningless ciphertext, depending on the architecture.

Integrity Protection and Anti-Replay

To defend against a malicious kernel or physical attacker modifying ciphertext in DRAM or replaying older encrypted states, advanced TEEs add integrity verification. They maintain a Merkle tree (or hash tree) of memory hashes stored in protected memory, with the root hash kept inside the CPU.

When a cache line is fetched from DRAM, the hardware computes its hash and verifies it up the tree to the root. If an attacker alters a single bit in DRAM or swaps a page with an older version, the hash verification fails and the CPU halts execution to prevent data corruption.

Attestation and Secure Provisioning

Before a user or remote server sends sensitive data into an enclave, it must verify that the enclave is genuine and running on real hardware. TEEs solve this through cryptographic attestation.

During enclave creation, the CPU measures the initial code and configuration by computing a cryptographic hash, often represented as \(M = H(\text{code} \parallel \text{config})\). The CPU signs this measurement using a hardware-embedded private key known as an endorsement key. A remote verifier checks this signature against the CPU manufacturer's public key, confirming that the enclave has not been tampered with before establishing an encrypted communication channel directly with the enclave.