AV1 Quantization Matrices Across Transform Blocks

The AOMedia Video 1 (AV1) codec utilizes frequency-dependent quantization matrices (QM) to optimize perceptual video quality across a wide range of transform block sizes. This article explains how AV1 defines, signals, and applies these scaling matrices to square and non-square transform blocks, adjusting quantization step sizes across spatial frequencies to preserve visual fidelity where human vision is most sensitive while discarding imperceptible high-frequency detail.

Frame-Level Signaling and Matrix Levels

AV1 does not transmit explicit, custom matrix tables within the bitstream to minimize overhead. Instead, it relies on a standardized set of predefined, hardcoded quantization matrices.

The bitstream signals quantization matrix usage at the frame header level:

• Enable Flag: A frame-level flag signals whether quantization matrices are active for the current frame.
• QM Levels: If enabled, the codec transmits separate matrix intensity levels for the luma (\(Y\)), chroma \(U\), and chroma \(V\) planes.
• Scaling Profiles: The levels range from 0 to 15. A value of 0 indicates a flat matrix (standard scalar quantization where all transform coefficients use identical step sizes). Values from 1 to 15 apply progressively steeper weighting, meaning high-frequency coefficients are quantized significantly more aggressively than low-frequency coefficients.

Support for Varied Transform Sizes and Shapes

AV1 supports multiple transform sizes ranging from 4x4 to 64x64, encompassing both square and rectangular aspect ratios (including 1:2, 2:1, 1:4, and 4:1 shapes, such as 4x8, 8x4, 16x4, and 8x32).

To support this variety without inflating decoder memory, AV1 standardizes base 2D matrix definitions derived from perceptual contrast sensitivity functions (CSF). The matrices correspond directly to the supported 2D transform dimensions:

• Square Blocks: Standard matrices exist for dimensions \(4\times4\), \(8\times8\), \(16\times16\), \(32\times32\), and \(64\times64\). In larger blocks (such as \(64\times64\)), high frequencies above a 32-coefficient cutoff are zeroed out by the transform process, and the matrix values map accordingly to the active coefficient area.
• Rectangular Blocks: For non-square blocks, AV1 selects predefined matrices tailored to the specific width-to-height ratio. These tables align with the anisotropic frequency distribution of rectangular transforms, ensuring that directional frequency scaling matches the unequal spatial resolutions along the horizontal and vertical axes.

Application to Transform Coefficients

During forward quantization in the encoder, the effective quantization step size for a coefficient at coordinate \((x, y)\) is computed as a product of the base frame quantizer and the scaling factor derived from the matrix:

1. Matrix Lookup: The encoder and decoder look up the weight \(M(x, y)\) corresponding to the block’s shape, size, component plane, and signaled QM level.
2. Coefficient Scaling: The base dequantization/quantization step size \(Q_{base}\) is modified by the matrix factor: \[\text{StepSize}_{effective}(x, y) = \frac{Q_{base} \times M(x, y)}{\text{Normalization Factor}}\]
3. Weight Distribution: The DC coefficient \((0, 0)\) and immediate low-frequency AC coefficients receive the lowest scale factors (closest to the base quantizer), preserving structural details, edges, and smooth gradients. Higher-frequency coordinates receive progressively larger divisors, increasing the likelihood that small, visually redundant coefficients round to zero.

Through this systematic lookup and scaling process, AV1 applies adaptive, frequency-dependent quantization across all valid transform configurations while keeping signaling overhead limited to a few bits per frame.