How Does the No-Execute Bit Prevent Stack Code Execution?

The No-Execute bit and the Write-XOR-Execute security policy protect classical computers by separating memory regions used for storing data from memory regions used for running instructions. By marking data stacks as non-executable in the system's page tables, the processor hardware prevents attackers from running malicious code injected through vulnerabilities like buffer overflows.

Understanding Memory Permissions and Page Tables

In modern operating systems, physical memory is managed through virtual memory, which divides address spaces into fixed-size blocks called pages. The operating system maintains data structures known as page tables to map virtual addresses to physical addresses and to store access permissions for each page.

Historically, early processor architectures only distinguished between pages that were readable and pages that were writable. If a page could be read, the processor assumed it could also be executed as machine code. This design meant that data areas, such as the call stack and the heap, were implicitly executable.

How Stack-Based Code Injection Works

A classic stack-based buffer overflow occurs when a program writes more data into a stack-allocated buffer than it can hold. This excess data overwrites adjacent values on the stack, including the function's saved return address.

In a traditional code injection attack, an attacker places malicious machine instructions—often called shellcode—directly into the buffer on the stack. The attacker then overwrites the return address so that when the current function finishes, the CPU's instruction pointer jumps to the injected bytes on the stack and executes them with the privileges of the running program.

The No-Execute (NX) Bit

To close this architectural gap, CPU manufacturers introduced the No-Execute (NX) bit—also known as the XD (Execute Disable) bit on Intel processors or the XN (Execute Never) bit on ARM architectures—as a dedicated flag in each page table entry.

When the NX bit is set on a memory page, the operating system is signaling to the processor's Memory Management Unit (MMU) that the page contains data, not executable instructions. Whenever the CPU attempts to fetch an instruction from a memory address, the MMU checks the corresponding page table entry. If the NX bit is active, the processor refuses to execute the instruction and immediately raises a hardware exception, typically a page fault. The operating system catches this fault and terminates the offending process, halting the attack before any injected code can run.

The Write-XOR-Execute (\(W \oplus X\)) Policy

The NX bit provides the hardware mechanism, while Write-XOR-Execute (\(W \oplus X\)) is the operating system policy that enforces it. Under \(W \oplus X\), every page in a process's address space may be either writable or executable, but never both at the same time.

In Boolean logic, this relationship is expressed using the exclusive OR (\(\oplus\)) operation:

\[W \oplus X = 1\]

Here, \(W\) represents write permission and \(X\) represents execute permission. A page containing program code is marked as readable and executable (\(W = 0, X = 1\)), preventing attackers from modifying the instructions. Conversely, a page containing the stack or heap is marked as readable and writable (\(W = 1, X = 0\)), preventing the CPU from executing any data placed there.

Limitations and Modern Context

While the NX bit and \(W \oplus X\) effectively stop direct code injection on the stack, they do not prevent all control-flow hijacking attacks. Attackers adapted by developing techniques such as Return-Oriented Programming (ROP), where they overwrite return addresses to chain together existing, legitimate executable instructions—called gadgets—already present in the program's code segment.

Additionally, certain legitimate software, such as Just-In-Time (JIT) compilers used in web browsers and language runtimes, must dynamically generate and execute machine code. To comply with \(W \oplus X\), these programs allocate memory as writable, write the compiled instructions, and then explicitly request the operating system to flip the page permissions from writable to executable before running the code. Despite these edge cases, hardware-enforced non-executable stacks remain a foundational defense in modern classical computing.