How ThreadSanitizer Detects Data Races in Chromium

This article provides a comprehensive overview of ThreadSanitizer (TSan) and its critical role in maintaining the stability and security of the Chromium web browser. It covers the fundamentals of what data races are, the mechanics of how ThreadSanitizer detects these concurrency bugs, and the exact workflows Chromium engineers use to integrate this tool into their local development, automated testing, and continuous integration pipelines.

What is a Data Race?

Before understanding ThreadSanitizer, it is necessary to understand the problem it solves. A data race occurs in a multithreaded program when two or more threads access the same memory location concurrently, at least one of these accesses is a write operation, and the threads are not using any mutual exclusion locks or synchronization primitives to control their access.

Data races lead to undefined behavior. Because thread scheduling is unpredictable, data races are notoriously difficult to reproduce and debug. In a complex application like a web browser, a data race can result in random crashes, silent data corruption, or severe security vulnerabilities that attackers can exploit.

What is ThreadSanitizer?

ThreadSanitizer (TSan) is a dynamic analysis tool built into the Clang and GCC compilers designed to detect data races in C++ and Go code. Unlike static analysis tools that merely guess where bugs might happen by reading source code, TSan monitors the program as it actually runs.

TSan consists of two parts:

  • Compile-time instrumentation: When code is compiled with TSan enabled, the compiler inserts special function calls before every memory read and write, as well as around synchronization events like mutex locks and thread creations.
  • A runtime library: As the program runs, this library intercepts those inserted calls. It uses a technique called "shadow memory" to keep a history of which threads have recently accessed specific memory addresses and whether those accesses were reads or writes.

If the runtime library sees that a thread is writing to a memory address that another thread recently accessed without a synchronization event (like a lock release and acquisition) happening in between, TSan immediately halts the program and outputs a warning report.

Why Chromium Needs ThreadSanitizer

The Chromium browser is a massively concurrent application. To keep the user interface responsive while loading complex web pages, Chromium splits its work across multiple processes, and each process runs dozens of threads. For example, the browser process has dedicated threads for the UI, disk caching, and network Input/Output (IO). The renderer process uses separate threads for executing JavaScript, compositing graphics, and handling user input.

With millions of lines of C++ code running across all these threads simultaneously, human error is inevitable. Engineers might accidentally read a variable on the UI thread while it is being modified on the IO thread. TSan is the primary defense against these mistakes.

How Chromium Engineers Use ThreadSanitizer

Chromium engineers integrate TSan into every stage of the software development lifecycle, from writing code locally to running massive automated servers.

1. Local Compilation and Testing

When an engineer is writing multithreaded code or investigating a suspected concurrency bug, they can build a special version of Chromium on their local Linux machine. Chromium uses the GN meta-build system. To enable TSan, an engineer simply adds the is_tsan = true flag to their build configuration.

Because TSan tracks every memory access, a TSan-enabled build of Chromium runs roughly 5 to 15 times slower than a normal build and uses up to 10 times more memory. Therefore, engineers usually do not run the full browser with TSan locally; instead, they compile and run specific, targeted unit tests or browser tests related to the code they are changing.

2. The Commit Queue and Continuous Integration

Chromium relies heavily on automated testing. Before any engineer's code change can be merged into the main Chromium codebase, it must pass through the Commit Queue (CQ).

The CQ includes dedicated clusters of Linux servers that continuously build Chromium with TSan enabled and run thousands of automated tests. If an engineer's new code introduces a data race that is triggered during any of these tests, TSan catches it, fails the test, and blocks the code from being merged.

3. Fuzzing with ClusterFuzz

Some data races only happen under bizarre, edge-case conditions that standard unit tests never check. To catch these, Chromium uses ClusterFuzz, a distributed fuzzing infrastructure. ClusterFuzz generates millions of random, malformed HTML pages, JavaScript files, and network inputs, feeding them into TSan-enabled builds of Chromium 24 hours a day. When a random input triggers a data race, the system automatically files a bug report and assigns it to the relevant engineering team.

Resolving Data Races in Chromium

When TSan detects a race, it generates a precise report. This report includes the exact memory address where the conflict occurred, the stack traces of both threads at the exact moment they touched the memory, and the stack trace of where that memory was originally allocated.

Equipped with this report, a Chromium engineer usually resolves the bug using one of the following methods:

  • Task Runners and Sequences: Chromium discourages the use of raw threads and locks. Instead, engineers fix races by refactoring the code to pass data asynchronously between threads using base::TaskRunner. Instead of two threads sharing memory, one thread packages the data into a task and posts it to the other thread's queue.
  • Sequence Checkers: To prevent future races, engineers add macros like DCHECK_CALLED_ON_VALID_SEQUENCE to their classes. This acts as an assertion that crashes the browser in debug mode if a class is ever accessed from the wrong thread, catching the mistake before TSan is even needed.
  • Locks and Atomics: If memory absolutely must be shared simultaneously for performance reasons, engineers will wrap the offending variable in a base::Lock or change the variable to a thread-safe std::atomic type, both of which TSan recognizes as valid synchronization.
  • Suppressions: Occasionally, TSan detects a data race in a third-party library that Chromium uses, but that Chromium engineers do not directly control. To prevent these external bugs from halting all browser development, engineers can add the specific function name to a TSan suppression file, instructing the tool to ignore that specific race until the third-party library can be updated.