What CDEF Methods Are Used by libaom?

The libaom reference software implements the Constrained Directional Enhancement Filter (CDEF) to post-process reconstructed video frames and eliminate ringing artifacts while preserving sharp edge details. To achieve this, libaom uses a multi-step process combining directional estimation, dual-stage non-linear low-pass filtering, and aggressive encoder search optimizations. By operating on an \(8 \times 8\) block level, the framework ensures highly parallelizable operations optimized for modern processor architectures.

Direction Estimation and Search Before any filtering occurs, libaom determines the structural orientation of each \(8 \times 8\) block through a dedicated direction search mechanism (cdef_find_dir).

• The block is analyzed across eight possible spatial directions (0 through 7).
• For each direction, pixel values along corresponding lines are averaged.
• The encoder computes the variance or sum of squared differences (SSD) between the original block and the averaged representation.
• The direction that yields the lowest variance is selected as the primary edge direction.

Dual-Stage Non-Linear Low-Pass Filtering Once the direction is identified, libaom applies a conditional low-pass filter (cdef_filter_block) that dynamically restricts blurring across sharp boundaries. This filter uses two distinct components:

• Primary Filtering: Acts strictly along the identified block direction to smooth out ringing. It relies on specific primary strength and damping parameters signaled in the bitstream.
• Secondary Filtering: Acts at a 45-degree angle relative to the identified direction to handle remaining artifacts. It utilizes separate secondary strength and damping values.
• The Constraint Mechanism: A mathematical thresholding function ensures that if a neighboring pixel’s value differs from the target pixel by more than the allowed strength, its weight in the filter is heavily penalized or ignored. This prevents the blending of distinct objects.

Encoder-Side Performance Optimizations Because evaluating every potential filter combination for a frame introduces massive computational overhead, libaom employs several specialized acceleration methods:

• Subsampling and Row Reduction: The CDEF search function (cdef_seg_search) can utilize spatial row-subsampling to evaluate fewer pixels during the direction and strength optimization process.
• Reference Frame Analysis: The encoder leverages motion and strength data from previously coded reference frames to narrow down or predict the optimal filtering parameters.
• Skip-Block Omission: Blocks that contain no prediction residual (skip blocks) bypass the filtering process entirely, significantly reducing CPU cycles.