Automated CI Testing for Open-Source AV1 Encoders

Testing AV1 encoders requires balancing software stability, encoding speed, and complex rate-distortion performance across vast sets of video data. The open-source community accomplishes this by integrating continuous automated testing pipelines into code repositories to run standard unit tests, bitstream conformance verifications, and objective visual quality evaluations on every code revision. By leveraging distributed compute clusters, specialized test harnesses, and standardized metrics like BD-rate and VMAF, developers detect algorithmic regressions, ensure specification compliance, and track compression efficiency gains in real time.

Automated Test Harnesses: The Role of AWCY

The primary tool used for testing open-source AV1 encoders (such as libaom, rav1e, and SVT-AV1) is "Are We Compressed Yet?" (AWCY), an open-source benchmarking service originally developed by Xiph.Org and Mozilla.

When a developer submits a pull request, CI triggers can send a build to AWCY. The system distributes encoding workloads across an array of worker nodes. These nodes encode standardized video test sets using both the target commit and a baseline branch (typically the master or main branch), comparing the results to quantify exactly how changes impact encoding speed and video quality.

Common Test Conditions (CTC) and Video Sets

To make benchmarks statistically meaningful, the community relies on Common Test Conditions defined by the Alliance for Open Media (AOMedia). CI pipelines execute tests using standard video sequences categorized by resolution, frame rate, and dynamic range:

• objective-1-fast: A short, diverse clip set used for rapid smoke tests on individual pull requests.
• Full CTC Sets: Comprehensive test suites containing 4K, 1080p, 720p, and standard-definition footage, typically run nightly or prior to major releases due to the high computational cost of full-depth encoding.

Measuring Rate-Distortion with Objective Metrics

Continuous integration for video encoders goes beyond simple pass/fail assertions. It must measure compression efficiency via rate-distortion (RD) performance. Automated pipelines parse outputs to compute:

1. Visual Quality Scores: Traditional metrics like PSNR and SSIM are calculated alongside perceptual models like VMAF (Video Multi-Method Assessment Fusion) and Butteraugli.
2. Bjøntegaard Delta Rate (BD-rate): Pipelines automatically plot RD curves across multiple quantization parameters (QPs) or target bitrates. The calculated BD-rate shows the percentage of bitrate saved or added for an equivalent visual quality level. A negative BD-rate indicates improved efficiency, while a positive BD-rate flags a regression.

Bitstream Conformance and Decoder Interoperability

Producing an unreadable or non-compliant bitstream is a critical failure. Automated workflows validate generated streams through multiple steps:

• Reference Decoders: Automated tasks feed the newly encoded bitstreams into strictly compliant reference decoders like aomdec to ensure standard conformance.
• Alternative Decoders: Streams are tested against fast, independent decoders such as dav1d to ensure cross-decoder compatibility.
• Fuzzing and Sanitizers: Tools like Google’s OSS-Fuzz continuously test encoder inputs using Clang sanitizers (AddressSanitizer, MemorySanitizer, UndefinedBehaviorSanitizer) to catch memory leaks, out-of-bounds reads, and integer overflows.

Multi-Architecture and SIMD Regression Checks

AV1 encoders rely heavily on hand-written assembly optimizations (AVX2, AVX-512, ARM Neon) to make encoding speeds practical. Standard CI platforms (such as GitHub Actions and GitLab CI) automatically build and test code across multiple target architectures and operating systems. These jobs run deterministic unit tests that compare the output of optimized SIMD functions directly against C fallback implementations to ensure identical output and prevent architecture-specific corruption.