How ROP Bypasses Non-Executable Stack Protections

This article explains how Return-Oriented Programming (ROP) circumvents non-executable stack protections (such as Data Execution Prevention or NX/DEP) in applications compiled from the C programming language. It covers the mechanics of traditional buffer overflows, why marking memory pages as non-executable fails against code-reuse attacks, how attackers locate and chain small instruction sequences known as "gadgets," and how these gadgets are executed sequentially to gain arbitrary code execution.

The Limitation of Non-Executable Stacks (NX/DEP)

In traditional stack buffer overflow vulnerabilities in C, such as those caused by functions like strcpy() or gets(), an attacker writes past a local buffer to overwrite the saved frame pointer and return address on the call stack. Historically, the attacker placed custom shellcode directly on the stack and set the return address to point back to this injected payload.

To mitigate this, operating systems implemented Write XOR Execute (W^X), commonly referred to as Data Execution Prevention (DEP) or the No-Execute (NX) bit. This memory protection model ensures that memory pages marked as writable (such as the stack and heap) cannot be executed by the CPU. When an instruction pointer (EIP or RIP) is redirected to an address on the stack, the processor raises a memory access violation or segmentation fault, immediately halting the process.

The Concept of Code Reuse

Return-Oriented Programming neutralizes the NX protection by shifting the attacker's approach from code injection to code reuse. Because the application's executable text segment and its shared libraries (such as the standard C library, libc) must remain executable for normal program operation, their memory pages are marked as readable and executable.

Instead of injecting new instructions, an attacker reuses existing machine code instructions already resident in memory.

Structure and Role of Gadgets

The fundamental building blocks of ROP are "gadgets." A gadget is a short sequence of existing machine instructions—typically between one and four instructions—that ends with a ret (return) instruction.

Examples of typical x86_64 gadgets include:

  • pop rdi ; ret (used to load a value from the stack into the rdi register, which holds the first argument in standard 64-bit C calling conventions)
  • pop rsi ; ret (used to load the second function argument)
  • mov [rax], rdx ; ret (used to write data to an arbitrary memory location)
  • syscall ; ret (used to trigger a direct kernel system call)

Attackers search the compiled C binary or loaded libraries using automated tools to extract these sequences and their corresponding memory addresses.

Chaining Gadgets via the Stack

During a ROP attack, the stack is not used to hold executable code; it is used as a data storage vehicle that controls program flow. The attacker crafts a payload that overwrites the stack with a sequence of gadget addresses and their required parameters:

  1. Stack Corruption: The attacker exploits a memory corruption vulnerability to overwrite the original return address with the address of the first gadget.
  2. First Gadget Execution: When the vulnerable C function finishes and executes ret, the CPU pops the first gadget's address into the instruction pointer, jumping execution to that gadget.
  3. Parameter Loading: If the gadget contains pop instructions, the CPU takes the values placed immediately beneath the gadget address on the stack and moves them into CPU registers.
  4. Subsequent Returns: The gadget concludes with its own ret instruction. The CPU pops the next value off the stack, which points to the second gadget in the chain.

By repeating this cycle, the stack pointer (ESP/RSP) acts as a pseudo-instruction pointer, methodically stepping through each gadget in the chain without executing a single byte of memory marked as non-executable.

Achieving Arbitrary Execution

By chaining gadgets together, an attacker achieves Turing-complete execution. In practice, attackers typically structure ROP chains to achieve one of two primary outcomes:

  • Invoking High-Privilege Functions: The attacker arranges registers according to the platform's Application Binary Interface (ABI) and calls standard C functions such as system("/bin/sh") or execve().
  • Disabling Protections: The attacker chains gadgets to call memory management functions like mprotect() (on Linux) or VirtualProtect() (on Windows) to re-mark a section of memory containing shellcode as executable, subsequently jumping into the newly executable buffer.