AV1 Transform Types Beyond Standard DCT

The AOMedia Video 1 (AV1) video codec achieves superior compression efficiency over predecessor formats largely through its advanced transform framework. While legacy video coding standards rely almost exclusively on the standard Discrete Cosine Transform (DCT), AV1 introduces a flexible, hybrid transform toolkit that incorporates the Asymmetric Discrete Sine Transform (ADST), the Flipped ADST (FLIPADST), and the Identity Transform (IDTX). This article breaks down these alternative primary transforms, explains their mathematical and practical roles in video compression, and illustrates how AV1 combines them to dramatically reduce residual prediction error across both camera-captured video and synthetic screen content.

The Asymmetric Discrete Sine Transform (ADST)

In intra-frame video coding, pixel blocks are predicted using neighboring boundary pixels from previously decoded blocks (typically above and to the left). As a result, the prediction error (the residual) is not uniformly distributed: it is typically very small near the reconstructed boundary and grows larger as the distance from the reference edge increases.

The standard DCT assumes an even boundary condition, making it suboptimal for modeling such asymmetric, one-sided error distributions. The ADST explicitly models this boundary-to-interior variance. By forcing the basis functions to approach zero near the known reference edge, the ADST concentrates the residual energy into fewer coefficients, significantly improving compression performance for directional intra-prediction blocks.

The Flipped ADST (FLIPADST)

The FLIPADST is an inverse-ordered variant of the ADST. While the standard ADST assumes that the residual error grows from left-to-right or top-to-bottom, certain prediction modes create inverted residual behaviors—where the error is larger near the starting boundary and tapers off toward the opposite side, or where the reference originates from the opposite edge.

By reversing the basis vectors of the standard ADST, FLIPADST provides optimal energy compaction for these inverted residual distributions without requiring distinct architectural hardware, as it reuses the core ADST computational logic with inverted addressing.

The Identity Transform (IDTX)

The Identity Transform acts as a frequency-domain bypass. When IDTX is selected, the transform phase is skipped, leaving the residual samples directly in the spatial domain for quantization and entropy coding.

Traditional frequency-domain transforms like the DCT and ADST can cause severe ringing artifacts around sharp, high-contrast edges because high frequencies cannot be completely eliminated. IDTX is particularly effective for:

• Screen Content: Text, user interface elements, and computer-generated graphics where pixel transitions are abrupt step functions rather than smooth gradients.
• Sparse Residuals: Blocks where only a handful of non-zero pixel errors exist, rendering full-frequency decomposition inefficient.

2D Separable Transform Combinations

AV1 processes two-dimensional blocks by applying 1D transforms independently along the horizontal and vertical axes. By drawing from DCT, ADST, FLIPADST, and IDTX, AV1 forms a matrix of 16 possible 2D transform combinations for square and rectangular blocks:

• Horizontal / Vertical Pairings: An encoder can apply a DCT horizontally and an ADST vertically (or vice versa), matching directional gradients within the residual.
• Partial Identity: A block can use IDTX in one direction and DCT or ADST in the other (e.g., standard frequency coding across a horizontal line, but spatial identity coding down a sharp vertical border).
• Full Identity: IDTX applied both horizontally and vertically handles complex artificial patterns cleanly.

By dynamically testing and selecting the transform pair that yields the lowest rate-distortion cost, AV1 maximizes coding efficiency across natural textures, directional edges, and synthetic content.