Understanding the Chromium Gardener Rotation System

This article explains how the Chromium project uses the Gardener rotation system to maintain the health of its continuous integration (CI) waterfall. It covers the core responsibilities of a Chromium Gardener, the revert-first policy used to keep the codebase stable, how the global rotation schedule is structured, and the specialized tools engineers use to monitor builds and triage failures in real time.

The Role of a Chromium Gardener

In the Chromium project, the continuous integration system builds and tests the browser across dozens of configurations—including Windows, macOS, Linux, Android, iOS, and ChromeOS. Because hundreds of changes land every day, broken builds or failing tests can quickly block thousands of engineers from submitting new code.

To prevent this, Chromium relies on "Gardeners" (a role formerly referred to as "Sheriffs"). A Gardener is an engineer temporarily assigned to monitor the CI waterfall, diagnose build and test failures, and take immediate action to keep the main development branch—known as "the tree"—green and open for commits.

Structure of the Rotation

The Gardener rotation is a shared responsibility among active Chromium committers and engineers from contributing organizations. The rotation is structured to maximize coverage and prevent burnout:

  • Follow-the-Sun Coverage: Shifts are divided across global time zones, primarily the Americas (AMER), Europe/Middle East/Africa (EMEA), and Asia-Pacific (APAC). This ensures that engineers only monitor the waterfall during their normal business hours while maintaining 24-hour coverage during the workweek.
  • Primary and Secondary Pairing: Shifts typically pair two engineers together—often an experienced Gardener with a newer one—to share the workload and distribute knowledge.
  • Specialized Rotations: Because Chromium is massive, a single rotation cannot cover everything. Beyond the main Chromium Trunk Gardeners, there are specialized rotations for specific platforms and components, such as Android, iOS, ChromeOS, GPU, V8 (JavaScript engine), and Performance.

Tools for Monitoring the Waterfall

Gardeners do not manually watch hundreds of individual buildbots on the raw CI waterfall. Instead, they rely on automated aggregation and alerting tools built on top of the LUCI (Layered Universal Continuous Integration) infrastructure:

  • Sheriff-o-Matic (SoM): This is the primary dashboard used by Gardeners. It automatically scans the waterfall, groups identical failures across different builders, and presents the Gardener with a prioritized list of actionable alerts.
  • LUCI Bisection: An automated system that runs in the background to pinpoint the exact changelist (CL) that introduced a compile failure or test regression, often proposing an automated revert before the Gardener even investigates.

The Triage Workflow: Revert First

When an alert appears on the dashboard, the Gardener follows a strict, fast-paced triage workflow. The guiding philosophy of the rotation is to prioritize the health of the tree over individual commits.

  1. Investigate the Failure: The Gardener checks the failure logs to determine if the issue is a compile failure, a consistent test failure, a flaky test (a test that passes and fails intermittently without code changes), or an infrastructure outage.
  2. Identify the Culprit: Using the builder's blamelist and automated bisection tools, the Gardener identifies the specific CL that caused the breakage.
  3. Revert the Changelist: Gardeners do not attempt to fix broken code themselves, nor do they wait for the original author to write a fix. They immediately revert the offending CL. This "revert-first" policy ensures the tree returns to a working state within minutes rather than hours.
  4. Disable Flaky Tests: If a test fails intermittently and the culprit CL cannot be easily identified or reverted, the Gardener will submit a quick change to disable the test in the codebase and file a bug report assigned to the test's owner.

Managing Tree Closures and Handoffs

If a severe breakage occurs—such as a compilation failure on a major platform—the tree closes automatically. When the tree is closed, the commit queue halts, and no engineer can land new code. Resolving tree closures is the Gardener's highest priority. Once the offending CL is reverted and the builder passes again, the tree reopens automatically.

At the end of their shift, the outgoing Gardeners write a concise handoff message in the dedicated Gardener chat channel. This summary informs the incoming rotation in the next time zone about any ongoing infrastructure issues, pending reverts, or flaky tests that are still being monitored, ensuring a seamless transition.