Neural Network In-Loop Filtering in Post-AV1 Codecs

As the Alliance for Open Media (AOMedia) advances research into next-generation video compression beyond AV1—commonly referred to within the AOM Video Model (AVM) research codebase—neural-network-based in-loop filtering has emerged as a cornerstone technology. This article examines the critical function of neural network in-loop filters (NNLF) in post-AV1 development, detailing how deep learning replaces or enhances traditional filtering stages, the mechanisms behind its compression efficiency, and the computational hurdles standard-setting bodies must overcome before commercial deployment.

The Function of In-Loop Filters in Video Codecs

In-loop filters are processing stages applied inside the reconstruction loop of a video encoder and decoder. Because they are "in-loop," the filtered frames serve as reference pictures for subsequent inter-frame prediction, directly influencing the accuracy and visual fidelity of future frames.

AV1 utilizes three primary heuristic and model-based in-loop filters:

1. Deblocking Filter (DF): Smooths boundaries caused by block-based transform and quantization partitions.
2. Constrained Directional Enhancement Filter (CDEF): Identifies directional edge patterns to reduce ringing artifacts without blurring critical borders.
3. Loop Restoration (LR): Employs Wiener filtering or Self-Guided restoration to recover lost edge details based on frame-level Wiener coefficients.

While effective, these filters rely on hand-crafted mathematical algorithms that operate within narrow, localized spatial contexts and struggle to generalize across diverse, complex compression artifacts.

The Neural-Network Paradigm Shift in Post-AV1 Research

In post-AV1 exploration, deep convolutional neural networks (CNNs) and lightweight attention mechanisms are deployed to augment or replace the traditional DF, CDEF, and LR pipelines. Instead of calculating static edge vectors or applying basic cross-shaped Wiener kernels, an NNLF takes reconstructed frames along with auxiliary side-information—such as partition trees, prediction modes, and quantization parameters (QP)—to predict and reconstruct a cleaned version of the frame.

Because neural networks model non-linear image degradation more accurately than classical algorithms, they achieve substantial gains in coding efficiency. In ongoing AVM evaluations, NNLF configurations typically deliver Bjøntegaard-Delta rate (BD-rate) savings between 5% and 15% across common test conditions. This means post-AV1 codecs can deliver the same perceived video quality as AV1 while using significantly less network bandwidth.

Improving Motion Compensation and Prediction

Beyond improving individual frame quality, the defining advantage of an in-loop neural filter is its compounding effect across temporal sequences. When reference frames stored in the Decoded Picture Buffer (DPB) have fewer artifacts and sharper edge structures, subsequent motion estimation and compensation algorithms yield smaller residual errors. Consequently, fewer transform coefficients must be transmitted for future inter-frames, compounding compression gains across long Groups of Pictures (GOPs).

Technical Challenges and Standardization Hurdles

Despite clear compression benefits, integrating neural networks into a global open standard introduces several engineering constraints:

• Computational Complexity: Deep neural networks demand orders of magnitude more arithmetic operations (MACs) than classical filters. Real-time decoding on consumer hardware requires extremely compact architectures.
• Deterministic Inference and Drift: Standardized video codecs require that every conforming decoder produces bit-identical output. Variations in floating-point calculations across different GPUs, NPUs, and CPUs can cause reference-frame drift, leading to severe visual corruption over time. Post-AV1 research actively focuses on strictly constrained integer arithmetic (such as 8-bit or 16-bit fixed-point quantization) to ensure deterministic results.
• Memory Bandwidth: Accessing multi-scale feature maps in deep networks consumes substantial memory bandwidth, which is often the primary bottleneck in mobile System-on-Chip (SoC) architectures.

Future Outlook

Neural-network in-loop filtering represents the transition from purely mathematical signal processing to hybrid deep-learning video coding. As research on post-AV1 formats matures, standardization efforts focus heavily on structural pruning, parameter-sharing, and hardware-conscious network topologies to ensure that the substantial coding gains of neural filters can be realized in practical, energy-efficient silicon.