AV1 Decoder Compliance Test Suites by AOMedia

This article provides an overview of the official compliance and validation test suites developed by the Alliance for Open Media (AOMedia) to verify AV1 decoder implementations. It outlines the core components of the AV1 Conformance Test Suite, the categories of test vectors used to validate profiles and levels, and the methodology employed to guarantee that third-party hardware and software decoders strictly follow the AV1 bitstream specification.

The Role of AOMedia Compliance Testing

AOMedia provides a comprehensive conformance framework to ensure that any compliant AV1 decoder can correctly parse and reconstruct pictures from valid AV1 bitstreams. Conformance testing prevents fragmentation across hardware implementations (such as GPUs, SoCs, and smart TVs) and software decoders by establishing an authoritative baseline for bitstream compliance and output fidelity.

The AV1 Conformance Bitstream Suite

The core of AOMedia’s validation toolkit is the AV1 Conformance Test Suite, a curated collection of bitstreams (.ivf or .obu files) paired with reference outputs (raw YUV/Y4M data and cryptographic MD5/SHA checksums). These vectors test every normative feature defined in the AV1 specification, including:

• Profile Coverage: Validation streams spanning the Main Profile (8-bit and 10-bit YUV 4:2:0), High Profile (adds YUV 4:4:4), and Professional Profile (adds YUV 4:2:2 and 12-bit depth support).
• Level and Tier Constraints: Bitstreams designed to operate at the exact boundary conditions of defined operational levels (from Level 2.0 up to Level 6.3), verifying maximum sample rates, bitrate caps, buffer sizes, and frame resolutions.
• Coding Tool Verification: Individual streams focused on distinct AV1 coding tools, including Constrained Directional Enhancement Filtering (CDEF), Loop Restoration Filters (Wiener and self-guided), Super-Resolution, Film Grain Synthesis, and Warp Motion/Global Motion vectors.
• Tiling and Scalability: Test sets validating uniform and non-uniform tile configurations, tile groups, and temporal/spatial scalability structures defined by Operating Points.

Types of Validation Vectors

AOMedia categorizes conformance streams into distinct groups based on their testing objective:

1. Functional Conformance Vectors: Standard bitstreams testing individual tools and syntax elements under typical operating conditions.
2. Stress and Boundary Vectors: Bitstreams crafted to hit maximum syntactic values (e.g., maximum motion vector ranges, peak transform sizes, extreme quantization steps) to ensure hardware pipelines do not overflow or bottleneck.
3. Negative and Error-Handling Vectors: Bitstreams containing out-of-spec parameters or malformed headers. While decoders are not required to decode invalid streams, these vectors verify that implementations reject or safely drop invalid bitstreams without crashing or entering undefined execution states.

The Reference Decoder as the Golden Standard

Compliance validation relies on libaom, AOMedia’s open-source reference software library. Within this library, the reference decoder (aomdec) acts as the "golden model."

To validate a target decoder:

• The target decoder parses the conformance bitstreams.
• It generates uncompressed YUV output frames.
• The generated frames are compared pixel-for-pixel against the output of the reference decoder.

For normative tools requiring exact reconstruction (such as arithmetic entropy decoding and inverse transforms), pixel-perfect matching is enforced via cryptographic checksum verification. For features where exact floating-point determinism may vary depending on implementation (such as certain post-processing or color space transformations), acceptable tolerance thresholds are specified by the working group.