AV1 Algorithms Slated for Overhaul in AV2

As the Alliance for Open Media (AOMedia) develops AV2 to succeed the AV1 video standard, researchers are systematically re-engineering the baseline coding tools to achieve substantial bitrate reductions. While AV1 combined technologies from VP9, Daala, and Thor, several algorithmic candidates reached their practical limits or suffered from high computational complexity. In AV2, key coding structures—most notably block partitioning, intra and inter prediction modes, transform toolsets, and in-loop restoration filters—are slated for comprehensive overhauls to push compression efficiency beyond modern standards like VVC.

Block Partitioning and Superblock Architecture

AV1 relies on a recursive superblock partitioning scheme (supporting 128×128 and 64×64 trees) with 10 partition types, including 4:1 and 1:4 rectangular splits. While flexible, this approach creates substantial decision complexity and lacks the spatial adaptability found in competing codecs.

In AV2, the partitioning engine is being overhauled with more versatile Multi-Type Tree (MTT) and Quad-Tree plus Multi-Type Tree (QTMT) structures. These algorithms eliminate rigid recursive tree constraints by allowing ternary, asymmetric, and independent luma-chroma split trees. This structural change targets AV1's inability to isolate high-frequency edges within fine-grained boundaries without incurring excessive split-signaling overhead.

Intra-Frame Prediction Enhancements

AV1 features 56 directional intra prediction angles paired with angle-delta signaling, smooth predictors, and Chroma from Luma (CfL) prediction. Although effective, AV1's intra prediction relies heavily on planar spatial continuity and struggles with complex local textures.

AV2 candidates overhaul this toolset in several distinct ways:

• Multi-Reference Line Prediction (MRLP): Replacing AV1’s single-line reference boundary, this technique allows predictors to sample reconstructed pixel rows and columns located further away from the target block.
• Matrix-Based Intra Prediction (MIP): Neural-inspired linear transformations replace standard directional interpolation for complex geometries.
• Expanded CfL and Cross-Component Tools: Cross-component linear models are being overhauled to support non-linear relationships and multi-model approximations, substantially improving chroma compression relative to AV1's simple affine scaling.

Inter Prediction, Affine Modeling, and OBMC

AV1 introduced Overlapped Block Motion Compensation (OBMC) and warped motion (affine prediction) alongside traditional motion vector differences. However, the implementation within AV1 is heavily constrained: warped motion is restricted to 8-parameter global models or local block-based approximations with fixed sub-sampling, and OBMC is limited to causal neighboring blocks.

For AV2, the inter-prediction pipeline is undergoing major revisions:

• Higher-Precision Motion Vectors: Migration toward 1/16-pel motion vector accuracy to improve temporal continuity in detailed scenes.
• Parametric Affine Motion Compensation: Transitioning from AV1’s limited warp mechanics to explicit 4-parameter and 6-parameter affine models that better capture zooming, rotation, and shear.
• Advanced Bi-Prediction and Blend Modes: AV1's Compound Inter-Intra and Wedge prediction tools are being overhauled with adaptive weighting derivation (such as generalized bi-prediction), reducing the signaling overhead required to blend multi-frame references.

Transform Core and Non-Separable Transforms

AV1 uses up to 16 transform combinations comprising 1D Discrete Cosine Transforms (DCT) and Asymmetric Discrete Sine Transforms (ADST), ranging from 4×4 to 64×64. A known bottleneck in AV1 is its exclusive reliance on separable 1D transforms, which leave residual statistical redundancies across diagonal pixel orientations.

In AV2, primary transforms are being augmented with Low-Frequency Non-Separable Transforms (LFNST). Applied as a secondary pass between the primary transform and quantization stages, LFNST decorrelates directional energy that AV1's separable DCT/ADST kernels fail to compact. Additionally, candidate toolsets include adaptive core transforms with dynamic matrix sizes, removing the hard truncation of high frequencies present in AV1’s 64×64 transform blocks.

In-Loop Filtering and Restoration Pipelines

The in-loop filtering pipeline in AV1 consists of three stages executed in sequence: the Deblocking Filter, the Constrained Directional Enhancement Filter (CDEF), and Loop Restoration (Wiener filter and Self-Guided filter). While CDEF and Wiener restoration delivered notable quality gains in AV1, their interaction is largely decoupled and incurs high computational complexity.

AV2 overhauls this pipeline by investigating:

• Unified Filtering Frameworks: Merging directional deringing and Wiener restoration into an integrated filtering pass to prevent redundant pixel modifications.
• Cross-Component Sample Adaptive Offset (CC-SAO): Introducing fine-grained sample classification across color planes to recover edge fidelity lost during quantization.
• Neural In-Loop Filtering: Experimental AV2 configurations test lightweight, integer-arithmetic convolutional neural network (CNN) loop filters designed to replace or supplement traditional CDEF passes.

Entropy Coding Engine

AV1’s entropy coder utilizes a multi-symbol arithmetic coding engine derived from Daala. While computationally efficient at higher bitrates, its probability adaptation mechanisms and context-modeling logic struggle with the highly dense syntax structures introduced by next-generation coding tools. AV2 algorithms refine context selection logic, introduce multi-hypothesis probability estimators, and optimize symbol throughput to accommodate the denser signaling profiles demanded by its upgraded prediction and transform tools.