How Telemetry Catches Speed Regressions in Chromium

This article explores how Telemetry, the performance testing framework for the Chromium web browser, simulates real-world user interactions to detect speed regressions. By combining automated user stories, deterministic network replays, and deep system tracing, Telemetry allows developers to measure how code changes impact real-world browsing performance across different platforms before those changes reach users.

Scripting Real-World User Stories

Telemetry defines its performance tests around the concept of "Stories." A story is a scripted scenario written in Python that dictates a specific sequence of user actions on a web page or a set of pages.

Instead of merely loading a static URL, a story simulates realistic human behaviors. These include scrolling down an infinite social media feed, clicking navigation buttons, typing into search bars, or performing pinch-to-zoom gestures on mobile devices. Telemetry uses specialized gesture APIs to execute these actions, ensuring that a simulated scroll or tap happens at the exact same speed and coordinates during every single test run.

Eliminating Variability with Web Page Replay

Testing browser performance against live websites is unreliable. Live content changes constantly, advertisements rotate, and network speeds fluctuate, making it impossible to know if a slowdown was caused by a bad code change in Chromium or a slow third-party server.

Telemetry solves this problem using a tool called Web Page Replay (WPR). When a developer creates a new user story, they record the interaction against the live website once. WPR captures all the HTML, JavaScript, CSS, images, and network responses, saving them into a static archive file.

When Telemetry runs the test later, it routes the browser’s traffic through a local WPR server using that archive. WPR even simulates specific network latencies and bandwidth constraints. This guarantees that the browser loads the exact same web page content under identical network conditions every time, isolating the browser's performance as the only variable.

Controlling the Browser via DevTools

To get accurate performance data, the testing framework must not interfere with the browser's normal operations. Telemetry does not require a specially modified build of Chromium, nor does it inject heavy testing code into the web pages being measured.

Instead, Telemetry controls a standard, unmodified Chromium binary remotely using the Chrome DevTools Protocol. Telemetry launches the browser, connects to its debugging port, and sends lightweight commands over a WebSocket connection to navigate to URLs and trigger the scripted user gestures. Because the browser is running in its natural state, the resulting performance metrics accurately reflect what a real user would experience on their device.

Measuring Metrics and Catching Regressions

While the browser executes the actions defined in a user story, Telemetry activates Chromium's internal tracing system. This system records a massive timeline of low-level events happening inside the browser, including JavaScript execution, DOM layout calculations, GPU rendering, and memory allocation.

Once the user story finishes, Telemetry stops the trace and processes the raw data into actionable metrics. Depending on the benchmark being run, it calculates values such as:

• Loading speed: Time to First Contentful Paint (FCP) and Largest Contentful Paint (LCP).
• Smoothness: Frame rates and dropped frames during scrolling or animations.
• Resource usage: Peak memory consumption, CPU time, and power drain.

These tests run continuously on dedicated hardware labs across Windows, Mac, Linux, Android, and ChromeOS as developers submit new code. The results are plotted on Chromium's performance dashboard. If a new code commit causes a metric in a user story to degrade beyond a strict statistical threshold, the system flags the speed regression, identifies the exact commit responsible, and alerts the engineering team to fix or revert the change.