GWP-ASan: Sampling Heap Allocations in Chromium Production

This article explains how the Chromium web browser utilizes GWP-ASan to detect memory corruption bugs in production environments without degrading user performance. It covers the probabilistic sampling mechanism used to intercept heap allocations, the guarded page architecture that traps spatial and temporal memory errors, and how the browser's crash handler extracts actionable debugging metadata when a memory fault occurs in the wild.

Probabilistic Sampling at the Allocator Level

Standard AddressSanitizer (ASan) introduces significant CPU and memory overhead, making it unusable in production releases of Chromium. GWP-ASan bypasses this limitation by only inspecting a tiny fraction of heap allocations.

In Chromium, GWP-ASan integrates directly into the memory allocator shim and PartitionAlloc. To avoid slowing down standard allocations, GWP-ASan uses a fast, thread-local counter. Every time the browser requests heap memory, the thread decrements its local counter. If the counter is greater than zero, the allocation proceeds normally through the default allocator.

When the counter reaches zero, the current allocation is selected for sampling. The counter is then reset to a new random number generated using a geometric distribution around a configured mean (for example, 1 in 1,000 allocations). This ensures that every allocation has an equal probability of being sampled while keeping the performance overhead of the check down to a single CPU instruction for non-sampled allocations.

The Guarded Slot Pool

Once an allocation is chosen for sampling, GWP-ASan diverts it away from the standard heap and into a pre-allocated, fixed-size region of virtual memory known as the guarded slot pool.

This pool is structured using operating system page-level permissions (mprotect on POSIX systems or VirtualProtect on Windows). It consists of a series of readable and writable memory pages—"slots"—separated by completely inaccessible "guard" pages. Because GWP-ASan relies on hardware page protections, it can only allocate memory in multiples of the system's page size (typically 4 KB).

To Limit physical memory overhead, Chromium configures GWP-ASan to use a small, fixed number of slots per process. If all slots are currently in use when a new allocation is sampled, the request simply falls back to the standard allocator.

Trapping Buffer Overflows and Underflows

When a sampled allocation is placed into an available slot, GWP-ASan must position it to catch out-of-bounds memory accesses. Because most allocations are smaller than a full 4 KB page, GWP-ASan randomly chooses to align the allocation to either the left edge or the right edge of the accessible page.

  • Left-aligned: The allocation starts at the very beginning of the slot, immediately adjacent to the preceding guard page. This configuration immediately traps buffer underflows.
  • Right-aligned: The allocation is pushed to the very end of the slot, placing its boundary directly against the following guard page. This configuration traps buffer overflows. (To maintain memory alignment requirements, right-aligned allocations may be padded by a few bytes, slightly reducing the chance of catching tiny overflows).

If the browser attempts to read or write past the boundary of the allocation and into a guard page, the CPU immediately triggers a hardware page fault, crashing the process at the exact instruction that caused the corruption.

Catching Use-After-Free Bugs

GWP-ASan is equally adept at catching temporal memory bugs, specifically Use-After-Free (UAF) errors.

When Chromium frees a sampled allocation, GWP-ASan intercepts the deallocation call. Instead of returning the memory to the standard heap, GWP-ASan makes a system call to mark the entire 4 KB slot as inaccessible.

If a dangling pointer within Chromium attempts to read from or write to that memory address after it has been freed, the CPU hits the newly restricted page and triggers an immediate page fault. To maximize the window of time available to catch a UAF bug, GWP-ASan places freed slots into a quarantine queue, delaying their reuse for future sampled allocations for as long as possible.

Metadata Recording and Crash Reporting

Trapping a memory bug in production is only useful if developers can diagnose the root cause. Every time GWP-ASan allocates or deallocates a guarded slot, it records crucial metadata in a separate, secure memory array. This metadata includes:

  • The thread ID that performed the operation.
  • The exact size and address of the allocation.
  • A compressed stack trace of the allocation call.
  • A compressed stack trace of the deallocation call (if freed).

When a GWP-ASan page fault occurs in the wild, Chromium's crash-reporting system, Crashpad, intercepts the exception. A specialized GWP-ASan hook inside Crashpad inspects the faulting memory address. If the address falls within the boundaries of the GWP-ASan guarded pool, the hook analyzes the state of the specific slot to classify the bug (e.g., Use-After-Free, Buffer Overflow, or Double Free).

Crashpad then bundles the recorded allocation and deallocation stack traces, along with the current crashing stack trace, into a minidump report and sends it to Chromium's crash servers. This provides engineers with a complete lifecycle of the corrupted memory, allowing them to fix complex bugs without ever needing to reproduce them locally.