How UBSan Detects Undefined Behavior in C++

Undefined Behavior Sanitizer (UBSan) is an analysis tool integrated into modern compilers like Clang and GCC to catch invalid operations during program execution. This article examines how UBSan functions under the hood, exploring how the compiler injects diagnostic assertions directly into intermediate code, how runtime libraries intercept violations, and how specific classes of undefined behavior—such as integer overflows, misaligned pointers, and invalid type casts—are caught before they cause silent memory corruption or security vulnerabilities.

Compile-Time Code Instrumentation

UBSan operates primarily at compile time, modifying the generated Intermediate Representation (IR) of a C++ program. When flags such as -fsanitize=undefined are enabled, the compiler inspects every abstract syntax tree (AST) node or IR instruction that could invoke undefined behavior according to the C++ standard.

Instead of generating raw, unchecked machine instructions, the compiler injects conditional branches around hazardous operations. For instance, when compiling an addition between two signed integers, the compiler does not simply emit an add instruction. Instead, it emits arithmetic intrinsics (such as llvm.sadd.with.overflow) that perform the addition while simultaneously monitoring the CPU's overflow flag. If an overflow is detected, the execution path branches to a specialized diagnostic routine; otherwise, the program continues normal execution.

The Role of the Runtime Library

The runtime component of UBSan acts as the handler for the checks inserted during compilation. When a generated condition detects an illegal state, execution jumps to an exported handler function (such as __ubsan_handle_add_overflow or __ubsan_handle_type_mismatch).

The compiler embeds static metadata at each check site, which is passed directly to the runtime handler. This metadata typically includes:

  • The exact source code location (file name, line number, and column number).
  • Type descriptions of the operands involved.
  • The specific violation category (e.g., division by zero, null pointer dereference).

Upon invocation, the runtime library parses this metadata alongside the dynamic runtime values of the operands. It constructs a descriptive error message and outputs it to stderr. By default, UBSan prints the diagnostics and attempts to resume program execution, but developers can configure it via -fno-sanitize-recover=undefined or environment variables to immediately abort execution or trigger a debugger breakpoint.

Detection Mechanisms by Category

Different types of undefined behavior require distinct runtime inspection strategies:

  • Pointer Alignment and Null Checks: Before a pointer is dereferenced, the compiler inserts a bitwise operation checking whether the memory address modulo the type's alignment requirement equals zero, and whether the address is null.
  • Array Bounds: When indexing into statically sized arrays or containers with known bounds, the compiler inserts explicit comparison instructions (index >= size) prior to the memory load or store.
  • Type Confusion and Downcasting: For polymorphic types, UBSan leverages C++ Run-Time Type Information (RTTI) and virtual table pointers (vptr). Before downcasting a base class pointer to a derived class, it verifies that the dynamic type recorded in the object's vtable actually matches or inherits from the target type.
  • Unreachable Code and Enums: When using enumeration types or reaching code paths designated as unreachable (such as std::unreachable()), the compiler inserts a check ensuring that the value matches a valid enum enumerator or unconditionally triggers a violation if an "unreachable" instruction is hit.

Performance Advantages Over Emulation

Unlike binary-translation tools such as Valgrind, which interpret instructions at runtime, UBSan introduces minimal performance overhead (often around 20% to 50%). Because the instrumentation happens directly inside the compiler's optimization pipeline, the optimizer can frequently eliminate redundant checks. If the compiler can prove through static analysis that a variable never exceeds a specific range, it removes the injected branch entirely, retaining full execution speed for provably safe code while strictly guarding uncertain operations.