How Web Platform Tests Track Safari Standards Compliance

This article provides an overview of how the Web Platform Tests (WPT) project uses automated testing pipelines to measure the standards compliance of the Safari web browser. By utilizing Apple's native WebDriver implementation, JavaScript assertion frameworks, and automated visual screenshot comparisons, WPT continuously evaluates Safari against official HTML, CSS, and JavaScript specifications. The resulting metrics are then compiled into public dashboards to track Safari's pass rates and overall web interoperability.

Automated Execution via WebDriver

To track Safari’s compliance without human intervention, the WPT infrastructure relies on continuous integration (CI) servers running macOS. These servers test both the stable release of Safari and Safari Technology Preview. WPT controls the browser using safaridriver, Apple's built-in implementation of the W3C WebDriver specification. The WPT runner uses safaridriver to launch the browser, navigate to specific test URLs hosted on a local test server, execute the tests, and record the outcomes.

Tracking HTML and JavaScript with testharness.js

For HTML specifications, DOM APIs, and JavaScript web integrations, WPT primarily relies on an internal framework called testharness.js. This framework runs directly inside Safari like a standard script.

• HTML Testing: The test runner loads an HTML document designed to trigger specific parsing behaviors, element states, or DOM manipulations. The script then queries the DOM to verify if Safari structured the document according to the WHATWG HTML specification.
• JavaScript Testing: Synchronous and asynchronous JavaScript tests execute API calls, Promises, and event loops. The harness uses assertion functions (such as assert_equals or assert_true) to compare Safari's actual output against the specification's expected result.

If an API requires a user gesture (like a mouse click or key press) to function due to browser security policies, WPT uses a companion script called testdriver.js. This script instructs safaridriver to simulate native, trusted user inputs so the test can proceed automatically.

Tracking CSS with Reference Tests

Because CSS dictates visual rendering, JavaScript assertions alone cannot verify if a layout or style is drawn correctly on the screen. To track CSS pass rates, WPT uses Reference Tests, commonly known as "reftests."

A reftest consists of two files: the test file (using the specific CSS feature being evaluated) and a reference file (using basic, universally supported HTML and CSS to create the exact same visual appearance). safaridriver loads the test file in Safari, takes a viewport screenshot, and then loads the reference file and takes a second screenshot. The WPT runner performs an automated pixel-by-pixel comparison of the two images. If the pixels match (or fail to match, in the case of a "mismatch" test), Safari passes the test. For non-visual CSS specifications, such as parsing rules and the CSS Object Model (CSSOM), WPT reverts to using testharness.js.

Calculating and Reporting the Pass Rate

As the test suite runs, a single test file may contain dozens of individual "subtests." Once a full run across the repository is complete, the WPT runner compiles the pass, fail, or timeout status of every test and subtest into a standardized JSON report.

This data is uploaded to the official WPT dashboard (wpt.fyi). The dashboard organizes the results based on the repository's folder structure, which mirrors official specification bodies (for example, /html, /css, and /webaudio). By dividing the number of passed tests by the total number of tests in each directory, the system tracks and displays Safari's exact standards pass rate over time.