AVIF Intra-Prediction Angle Selection Explained

AVIF utilizes the AV1 specification to compress still images by removing spatial redundancy within individual frames. A key mechanism in this process is directional intra-prediction, where pixel values in a target block are extrapolated from previously reconstructed boundary pixels along specific geometric angles. To maximize compression efficiency without causing prohibitive computational delay, AVIF encoders use a multi-stage selection algorithm combining structural analysis, fast cost estimation, and Rate-Distortion Optimization (RDO) to determine the ideal prediction angle.

The AV1 Angular Prediction Framework

AVIF supports 56 directional intra-prediction angles. These angles are structured around eight base "nominal" directional modes, which correspond to major geometric orientations (such as horizontal, vertical, and diagonals ranging from 45 to 203 degrees).

To achieve finer angular granularity, the encoder allows each nominal mode to be adjusted by an angle delta. This delta modifies the base trajectory by -3 to +3 steps in increments of approximately 3 degrees. Rather than performing a brute-force search over all 56 angles for every block, the encoder uses a tiered pruning process.

Step 1: Spatial Gradient and Edge Detection

Before running expensive mathematical transformations, the encoder analyzes the raw pixel data of the current block to detect structural orientation. By calculating horizontal and vertical gradients using Sobel filters or structure tensors, the encoder estimates the dominant edge direction of the block. Angles that strongly conflict with this edge direction are immediately pruned from the candidate list, significantly reducing the search space.

Step 2: Nominal Mode Evaluation (Coarse Search)

The encoder next evaluates the remaining nominal directional modes alongside non-directional modes (such as DC, Paeth, and Smooth).

To test these modes quickly:

• Boundary pixels from the top and left neighboring reconstructed blocks are projected into the target block along the base angles.
• Instead of running a full transform, quantization, and entropy coding loop, the encoder calculates a low-complexity error metric, typically the Sum of Absolute Transformed Differences (SATD) using a Hadamard transform.
• The modes producing the lowest SATD costs are retained as candidates for finer evaluation.

Step 3: Angle Delta Refinement (Fine Search)

Once the best nominal directional mode is identified, the encoder searches for the optimal sub-angle:

• The encoder evaluates the directional offsets (deltas from -3 to +3) around the winning nominal mode.
• Pixel projection at non-integer coordinates is resolved using sub-pixel interpolation (typically via a 2-tap bilinear filter).
• The encoder computes the prediction residual for each delta offset, again comparing the candidates using low-cost distortion metrics to narrow the selection down to the top one or two sub-angles.

Step 4: Full Rate-Distortion Optimization (RDO)

The final selection is made using Rate-Distortion Optimization. For the top candidate angles, the encoder performs the complete encoding pipeline:

1. Generates the residual block by subtracting the predicted block from the original source pixels.
2. Applies the forward transform (such as DCT or Asymmetric Discrete Sine Transform) and quantizes the coefficients.
3. Entropy codes the quantized coefficients and the syntax elements required to signal the nominal mode and angle delta.
4. Computes the Lagrangian cost: \(J = D + \lambda R\), where \(D\) represents pixel distortion (Sum of Squared Errors), \(R\) is the exact number of bits required to encode the mode and residual, and \(\lambda\) is the Lagrange multiplier tied to the quantization parameter (QP).

The angle that yields the lowest cost \(J\) is selected as the winning intra-prediction mode and written to the AVIF bitstream.