Subpel Interpolation Filter Selection in AV1

In video compression, sub-pixel (subpel) motion compensation enables motion vectors to point to fractional pixel locations, requiring interpolation filters to reconstruct intermediate sample values. The AV1 codec significantly enhances this process compared to older standards by introducing a flexible, switchable interpolation filtering mechanism that operates on both the frame and block levels. This article explains the available filter types in AV1, the dual-filter architecture allowing independent horizontal and vertical processing, and the encoder-side selection process driven by rate-distortion optimization (RDO).

Filter Types in AV1

AV1 defines four primary interpolation filter types, each tailored to different visual textures and frequency components:

1. Regular: An 8-tap filter with standard pass-band characteristics, optimized for natural motion and balanced frequency retention.
2. Smooth: An 8-tap filter that attenuates high frequencies. It is ideal for blurry, flat, or out-of-focus regions where reducing noise prevents artifact propagation.
3. Sharp: An 8-tap filter designed to retain and emphasize high-frequency edges and fine details, preventing blurring along sharp motion boundaries.
4. Bilinear: A simple 2-tap linear filter providing low computational complexity. It performs well in flat areas or where precise edge reconstruction is not rate-distortion optimal.

Dual Interpolation Filtering

Unlike predecessors such as VP9, which applied the same filter in both spatial directions, AV1 introduces Dual Interpolation Filtering. This allows an encoder to select horizontal and vertical filters independently for a single block.

Because motion and edge orientations often vary along the X and Y axes (such as horizontal panning across vertical blinds), decoupling the filters grants \(4 \times 4 = 16\) possible filter combinations per block. This directional flexibility substantially minimizes prediction error in complex scenes.

Signaling Hierarchy

The filter selection occurs within a two-tiered signaling structure:

• Frame Level: The frame header specifies whether a single global interpolation filter applies to all inter blocks in the frame or whether Switchable mode is enabled.
• Block Level: When the frame header enables switchable filtering, the bitstream signals the specific filter index for each inter prediction unit (or dual indices if horizontal and vertical filters diverge).

The Encoder Selection Process

During inter-prediction mode evaluation, the encoder selects the optimal filter using the following stages:

1. Motion Vector Refinement

Motion estimation initially finds integer and fractional motion vectors (at 1/8-pel precision in AV1). The encoder typically uses a default or symmetric filter (often the Regular 8-tap) during early subpel search steps to isolate candidate motion vectors.

2. Rate-Distortion Optimization (RDO)

Once candidate motion vectors are established, the encoder evaluates the performance of different interpolation filters. For each candidate filter or filter pair:

• The subpel samples are interpolated horizontally and vertically.
• The prediction residual is calculated against the original source block.
• The transform, quantization, and entropy coding stages are evaluated to determine the total bit cost (\(R\)) and distortion metric (\(D\)), such as Sum of Squared Errors (SSE).
• The RD cost is calculated using the Lagrangian formulation: \[J = D + \lambda \cdot R\] where \(\lambda\) represents the Lagrange multiplier tied to the frame's quantization parameter (QP).

The encoder chooses the filter or filter pair that produces the minimal RD cost \(J\).

3. Complexity Reduction and Fast Heuristics

Evaluating all 16 dual-filter permutations for every block partition is computationally prohibitive. Practical AV1 encoders (such as libaom or SVT-AV1) implement heuristic pruning strategies:

• Symmetric Testing: Encoders test symmetric pairings (e.g., Regular-Regular, Smooth-Smooth) first.
• Gradient-Based Pruning: Local gradient analysis detects whether vertical or horizontal edge structures dominate, skipping Sharp or Smooth tests when directional variance is low.
• Early Termination: If a fast Bilinear or Regular evaluation satisfies a strict RD threshold relative to the skip mode, further filter searches are bypassed.