AV1 CDEF Dominant Direction Calculation Formula

The Constrained Directional Enhancement Filter (CDEF) in the AV1 video codec identifies the primary edge orientation of an 8x8 block to apply directional smoothing without blurring sharp details. To determine this dominant direction, CDEF analyzes eight discrete angles and calculates which direction minimizes the mean squared error along directional lines. This article breaks down the mathematical formula and algorithmic steps used to compute directional energy, select the optimal angle, and determine directional confidence.

The 8 Discrete Directions

CDEF tests eight possible directions, indexed from \(d \in \{0, 1, \dots, 7\}\). These correspond to angles at increments of 22.5°:

• \(d = 0\): Horizontal (0°)
• \(d = 1\): 22.5°
• \(d = 2\): Diagonal (45°)
• \(d = 3\): 67.5°
• \(d = 4\): Vertical (90°)
• \(d = 5\): 112.5°
• \(d = 6\): Diagonal (135°)
• \(d = 7\): 157.5°

Mathematical Formulation

Within an 8x8 block of pixels, each direction \(d\) defines a set of parallel lines \(k\). Let \(x_{p}\) represent the value of pixel \(p\) belonging to line \(k\) for direction \(d\), and let \(N_{d,k}\) be the number of pixels located along that specific line.

The algorithm calculates the sum of pixel intensities along each line \(k\):

\[S_{d, k} = \sum_{p \in \text{line}(d, k)} x_p\]

The objective is to find the direction that minimizes the variance of pixels along the lines, which is mathematically equivalent to maximizing the variance between the lines. The directional energy \(E_d\) for direction \(d\) is computed as:

\[E_d = \sum_{k} \frac{S_{d, k}^2}{N_{d, k}}\]

The dominant direction \(d^*\) is the angle that maximizes this energy:

\[d^* = \arg\max_{d} E_d\]

Directional Variance and Confidence

After identifying the optimal direction \(d^*\), the codec computes a confidence metric to determine whether directional filtering is beneficial or if the block represents a flat or chaotic region.

The directional variance \(\sigma^2\) evaluates the contrast between the dominant direction and the average block energy:

\[\sigma^2 = E_{d^*} - \frac{1}{64} \left( \sum_{p \in \text{block}} x_p \right)^2\]

If \(\sigma^2\) is low, the block lacks a defined edge, and directional filtering strength is attenuated. If \(\sigma^2\) is high, filtering is applied primarily along direction \(d^*\).

Implementation Optimizations

In hardware and software implementations (such as libaom and rav1e):

1. Integer Division Avoidance: The division by \(N_{d,k}\) is structured using lookup tables and fixed-point arithmetic to avoid runtime division instructions.
2. Subsampling: Encoders often subsample pixel calculations (e.g., evaluating every second pixel in a checkerboard pattern) during direction search to reduce computational complexity while retaining direction accuracy.