Why CDEF Runs Before Loop Restoration in AV1

The AV1 video codec incorporates a sophisticated multi-stage in-loop filtering pipeline to suppress compression artifacts and restore high-frequency details. In this pipeline, the Constrained Directional Enhancement Filter (CDEF) runs immediately after the deblocking filter and directly before the Loop Restoration stage. This sequence is intentionally designed because CDEF suppresses severe directional ringing artifacts introduced during quantization, providing a cleaner base signal that enables the subsequent mathematical restoration filters to accurately restore genuine image details without amplifying noise.

The Role of CDEF

CDEF combines edge-directed deringing from the Daala codec with the directional filtering concept from the Thor codec. Its primary task is to eliminate ringing artifacts around sharp edges without blurring the edges themselves. It operates by identifying the primary direction of gradients within small blocks (typically 8x8) and applying a directional non-linear filter along those edges.

Because transform coding and coarse quantization generate high-frequency ringing that spreads across block boundaries, CDEF acts as a targeted artifact-suppression filter. It does not attempt to reconstruct lost textures; rather, it stabilizes edges and smooths out spurious oscillations.

The Role of Loop Restoration

Loop Restoration is the final stage of the in-loop filtering process. It consists of two complementary tools: a separable symmetric Wiener filter and a Self-Guided Restoration (SGR) filter.

Unlike CDEF, Loop Restoration operates by comparing the degraded frame to the original source frame at the encoder side. The encoder computes filter coefficients that mathematically minimize the Mean Squared Error (MSE) between the filtered reconstructed frame and the source. These coefficients are then transmitted in the bitstream so the decoder can reconstruct fine details and counteract the cumulative blurring caused by earlier processing steps.

Why CDEF Precedes Loop Restoration

  1. Preventing the Amplification of Ringing
    Wiener filters and self-guided filters act essentially as deblurring and sharpening mechanisms. If Loop Restoration were executed before CDEF, its sharpening filters would treat directional ringing artifacts as genuine image features, sharpening and exacerbating the visual distortions. Running CDEF first removes these artifacts, ensuring the restoration step only enhances true edge structures.

  2. Linear Optimization Over Cleaner Baselines
    Loop Restoration relies on linear or quasi-linear models to restore lost frequencies. Ringing artifacts caused by high-frequency transform coefficient truncation are inherently non-linear and context-dependent. CDEF’s non-linear, directional smoothing resolves these complex localized distortions. Once the non-linear errors are smoothed out, the linear MSE-minimization algorithms in Loop Restoration can converge more effectively toward the original signal.

  3. Maximized Coding Efficiency and Rate-Distortion Performance
    Because the encoder calculates Loop Restoration parameters based on the output of the preceding filters, providing Loop Restoration with a CDEF-filtered input significantly reduces the residual error. The resulting Wiener or SGR coefficients require fewer signaling bits to achieve a lower overall distortion, directly improving the codec's rate-distortion performance.

By structuring the AV1 pipeline to move from structural artifact removal (Deblocking and CDEF) to global detail synthesis (Loop Restoration), each filter addresses the specific degradation it is best suited to solve, yielding optimal visual quality and high compression efficiency.