AVIF Transforms Beyond Discrete Cosine Transform

The AV1 Image File Format (AVIF), derived from the video codec AV1 developed by the Alliance for Open Media, achieves superior compression efficiency over legacy formats like JPEG by modernizing how pixel data is mapped into the frequency domain. While traditional image formats rely almost exclusively on the standard Discrete Cosine Transform (DCT), AVIF introduces a flexible transform framework that incorporates the Discrete Sine Transform (DST), the Identity Transform (IDTX), and hybrid two-dimensional transform combinations tailored to directional intra-prediction and complex visual patterns.

The Discrete Sine Transform (DST)

In image compression, prediction algorithms estimate pixel values based on adjacent reconstructed blocks, leaving behind a prediction residual. The residual error often exhibits asymmetric energy distribution; for example, prediction error tends to be smallest near the reference boundary and increases farther into the block.

Standard DCT (specifically DCT-II) assumes symmetric boundary conditions and works best when residual energy is evenly spread. AVIF addresses this limitation by using the Discrete Sine Transform (specifically DST-VII). Mathematically, DST-VII enforces zero-value boundary conditions at the starting edge, making it optimal for representing residuals that grow monotonically along the prediction direction. By concentrating energy into fewer transform coefficients, DST reduces high-frequency noise and preserves fine details in directional gradients.

The Identity Transform (IDTX)

The Identity Transform bypasses frequency-domain conversion entirely along a given axis, effectively leaving the spatial residuals untouched prior to quantization.

Traditional frequency transforms like the DCT assume that visual signals are smooth and continuous. When applied to sharp artificial discontinuities—such as line art, UI elements, computer graphics, or text—DCT causes high-frequency oscillations known as ringing or Gibbs phenomenon. By employing the Identity Transform, AVIF quantizes spatial pixel values directly. This avoids blurring, eliminates halo artifacts around sharp edges, and dramatically improves compression ratios for screen-content images.

Asymmetric and Hybrid 2D Transform Combinations

AVIF does not restrict a block to using the same mathematical transform in both the horizontal and vertical directions. Instead, it treats 2D transforms as separable operations, allowing the encoder to pair different 1D transforms along the X and Y axes.

An encoder can choose from up to 16 combinations for a given block, mixing and matching:

• DCT / DCT: Standard symmetric frequency compaction.
• DCT / DST or DST / DCT: Ideal for blocks with directional intra-prediction along one axis and smooth variation along the other.
• IDTX / DCT or DCT / IDTX: Ideal for textures with vertical or horizontal stripes, scanlines, or straight boundaries.
• IDTX / IDTX: A complete bypass of frequency transforms in both dimensions for complex sharp patterns.

Non-Square and Large Transform Sizes

Beyond introducing new transform types, AVIF expands the physical geometry to which these transforms apply. Legacy JPEG restricts DCT operations to rigid 8×8 square blocks. AVIF supports transform sizes ranging from 4×4 up to 64×64 pixels, including non-square rectangular shapes with 1:2, 2:1, 1:4, and 4:1 aspect ratios (such as 4×16 or 32×8). Applying DST, DCT, and Identity transforms across rectangular partitions allows AVIF to match the natural geometry of edges and gradients far more accurately, minimizing residual energy before entropy coding.