AV1 Interpolation Filter Selection Explained

The AV1 video codec dynamically selects between bilinear, regular, and sharp interpolation filters to optimize sub-pixel motion compensation for each block. This selection is governed by the encoder’s Rate-Distortion Optimization (RDO) process, which evaluates image detail, frequency content, and signaling costs to determine which filter produces the highest visual fidelity at the lowest bitrate. By allowing independent horizontal and vertical filtering choices, AV1 balances edge sharpness, artifact suppression, and computational efficiency across diverse video content.

The Role of Each Filter Type

When predicting motion at fractional-pixel positions (half-pel or quarter-pel), the codec must interpolate samples from the reference frame. Each filter serves a distinct purpose based on frequency response:

• Regular Filter: An 8-tap filter designed with a balanced frequency response. It functions as the default standard for typical textures, smoothing out high-frequency noise while preserving standard edge definition.
• Sharp Filter: An 8-tap filter with extended high-frequency retention. It is tuned for sharp, high-contrast edges and fine textures, preventing the blurring that standard interpolation filters might introduce across fast-moving, detailed boundaries.
• Bilinear Filter: A simple 2-tap linear interpolation filter. While it offers lower frequency preservation compared to 8-tap variants, it introduces no ringing artifacts, uses significantly less computational overhead, and is optimal for flat, low-detail areas where higher-order taps yield no compression benefit.

Rate-Distortion Optimization (RDO)

The primary mechanism that decides which filter to apply is Rate-Distortion Optimization. During the motion estimation and mode decision phase, the encoder evaluates potential filter candidates using the rate-distortion cost formula:

\[\text{Cost} = \text{Distortion} + \lambda \times \text{Rate}\]

1. Distortion Evaluation: The encoder generates a motion-compensated prediction block using a candidate filter and calculates the error (typically Sum of Squared Errors or Sum of Absolute Transformed Differences) relative to the source block.
2. Rate Estimation: The encoder estimates the number of bits required to signal the choice of filter in the bitstream, alongside the motion vectors and residual data.
3. Selection: The filter that yields the lowest total cost is selected for the block. If the sharp filter preserves edge fidelity enough to significantly reduce residual energy, its distortion savings outweigh its signaling cost. If the block lacks fine detail, the regular or bilinear filter is chosen to avoid unnecessary overhead or ringing.

Dual-Filter Selection (Separable Filtering)

AV1 supports 2D separable interpolation filtering, meaning horizontal and vertical sub-pixel displacements can utilize different filters within the same prediction block. For instance, a block containing purely horizontal lines can apply a sharp filter vertically to maintain the edge contrast while using a regular or bilinear filter horizontally where high-frequency detail is absent. The encoder evaluates the combinations (e.g., Regular-Sharp, Sharp-Regular, Bilinear-Bilinear) during the RDO loop to find the optimal directional match.

Encoder Heuristics and Speed Presets

Evaluating every filter combination for every block is computationally expensive. Practical AV1 encoders, such as libaom and SVT-AV1, use algorithmic heuristics to streamline the decision:

• Gradient and Variance Analysis: Blocks with low spatial variance or low directional gradients bypass sharp filter evaluation entirely.
• Early Termination: If a standard regular filter test yields a distortion score below a specific threshold, the encoder skips the evaluation of alternative filters.
• Preset-Based Pruning: In faster encoding modes, switchable filtering is often disabled entirely, locking the encoder to the regular 8-tap filter or switching exclusively to bilinear filtering for fast-motion, low-complexity scenes to conserve encoding time.