dav1d AVX2 and AVX-512 Optimization Paths

This article examines the specialized AVX2 and AVX-512 instruction set architecture (ISA) optimization paths implemented in the open-source dav1d AV1 decoder. It details the specific algorithmic workloads targeted by these vector extensions—including inverse transforms, directional intra prediction, inter-frame motion compensation, in-loop filtering, and film grain synthesis—demonstrating how 256-bit and 512-bit vector pipelines maximize AV1 decode throughput on modern x86 architectures.

Architectural Overview

dav1d relies on handcrafted assembly routines using NASM to extract maximum performance from modern x86 CPUs. Rather than relying on auto-vectorization from C code, the decoder implements dedicated code paths for 8-bit and 10-bit/12-bit (High Bit Depth, or HBD) pixel pipelines.

• AVX2 Implementations: Utilize 256-bit wide vector registers (ymm), FMA3, and 256-bit integer extensions (AVX2). This represents the primary performance baseline for modern x86-64 processors.
• AVX-512 Implementations: Primarily target architectures supporting the AVX-512F (Foundation), AVX-512BW (Byte and Word instructions), AVX-512DQ (Doubleword and Quadword), and AVX-512VL (Vector Length extensions) instruction subsets. The presence of AVX-512BW is essential for dav1d, as pixel manipulation and intermediate transform coefficients rely extensively on 8-bit and 16-bit packed data structures.

Inverse Transforms (ITX)

The inverse transform stage converts frequency-domain coefficients back into spatial-domain residuals. AV1 supports multiple transform kernels—Discrete Cosine Transform (DCT), Asymmetric Discrete Sine Transform (ADST), FlipADST, and Identity (IDTX)—in block dimensions ranging from 4x4 up to 64x64.

• AVX2 Path: Processes rows and columns using 16-bit intermediate integer precision. For 8-bit video, intermediate stages use signed 16-bit values via vpmaddwd and vpackssdw. Blocks of size 16x16 and 32x32 leverage 256-bit registers to calculate 16 data points per instruction, transposing blocks in-register via vpunpcklwd, vpunpckhwd, and vpermq.
• AVX-512 Path: The 512-bit registers (zmm) double the processing width, allowing 32 parallel 16-bit butterfly operations per clock cycle. For large 32x32 and 64x64 transforms, AVX-512 eliminates loop overhead by keeping wider matrix states loaded in the 32 vector registers (zmm0–zmm31), significantly cutting down memory spills and reload latency.

In-Loop Filtering

AV1 employs three sequential in-loop filters to eliminate compression artifacts: the Deblocking Filter (LF), the Constrained Directional Enhancement Filter (CDEF), and the Loop Restoration Filter (LR).

Deblocking Filter (LF)

Deblocking operates across 4x4 block grid boundaries with threshold-based conditional clipping.

• AVX2: Utilizes vpminub/vpmaxub (or 16-bit equivalents for HBD) and vpcmpeqb to compute adaptive filter strengths across four 8-pixel or 16-pixel edges simultaneously.
• AVX-512: Employs AVX-512BW masking registers (k1–k7). Conditional checks yield bitmasks that drive vmovdqu8 or vpblendmb, removing vector blending bottlenecks and eliminating pipeline-stalling conditional branches.

Constrained Directional Enhancement Filter (CDEF)

CDEF identifies the primary edge direction across 8x8 blocks and applies a non-linear deringing filter along and across that direction.

• AVX2: Direction search calculates directional variance over multiple angles using 256-bit SAD (Sum of Absolute Differences) patterns. The filtering phase uses vectorized clamping via min/max instructions on packed data.
• AVX-512: Analyzes both the primary and secondary filtering taps for two 8x8 blocks in parallel inside a single 512-bit register. Ternary logic operations (vpternlogd) accelerate complex bitwise condition checks needed during variance calculation.

Loop Restoration (LR)

LR includes Wiener filtering (separable symmetric filtering) and Self-Guided Restoration (SGRproj).

• Wiener Filter: AVX2 utilizes horizontal and vertical filtering passes with symmetric 7-tap coefficients, computing the outer-product multiply-accumulate steps with vpmaddwd. AVX-512 accelerates this by processing 32 samples per instruction pass, fully saturating the compute pipeline.
• Self-Guided Restoration: SGRproj relies on box filters to compute local means and variances. AVX-512 provides high speedups through fused multiply-add (vpmaddwd and vpmulhrsw) operations across 512-bit registers to calculate integral images and standard deviations over large restoration units (typically 64x64 to 256x256 pixels).

Motion Compensation and Inter Prediction

Inter prediction generates predicted samples using motion vectors with subpixel accuracy (1/8th-pel precision), requiring 8-tap interpolation filters (Regular, Smooth, or Sharp).

• Subpixel Interpolation:
  • AVX2: Simultaneously filters 16 horizontal or vertical pixels using packed signed 8-bit or 16-bit weights. Horizontal filtering leverages vpshufb for efficient subpixel tap alignment prior to horizontal accumulation.
  • AVX-512: Evaluates 32 subpixel samples in a single instruction sequence. The larger register space allows storing all filter phases (interpolation sub-pel steps 0–15) directly inside the register file, eliminating cache loads for filter coefficients during subpixel shifts.
• Compound and Masked Blending: When combining two prediction surfaces (e.g., wedge-based or difference-modulated prediction), the AVX2 path processes 256-bit weighted averages. The AVX-512 path applies vpblendmb and vpmulhrsw to execute sample-level blending across full cache lines in single clock cycles.

Intra Prediction and Chroma from Luma (CfL)

Intra prediction synthesizes blocks based on adjacent boundary pixels.

• Directional Modes: Require projecting boundary samples across angles from 45 to 207 degrees. AVX2 vectors project and interpolate samples across 8-wide and 16-wide rows using vpermd and vpshufb. AVX-512 optimizes large intra blocks (32x32, 64x64) by calculating full intra-prediction fields across 32 columns concurrently.
• Chroma from Luma (CfL): Reconstructs AC chroma components from downsampled reconstructed luma pixels. AVX2 downsamples luma via horizontal averaging (vpavgb) and applies linear regression scaling. AVX-512 processes the luma-to-chroma mapping for full 32x32 chroma regions within four instruction dispatches.

Film Grain Synthesis

AV1 specifies an algorithmic film grain model, bypassing standard grain transmission by generating synthetic noise via an autoregressive (AR) process at playback.

• Noise Field Generation: dav1d implements a pseudo-random number generator and an AR filter over a predefined grain matrix.
• AVX2 vs. AVX-512 Optimization: The AR process is inherently serial in its spatial neighborhood dependencies, but the final additive blending stage is embarrassingly parallel. The AVX2 path generates and scales the grain patterns over 16-pixel chunks, while the AVX-512 path broadcasts the scale factor and calculates noise clipping/addition across 64 pixels per operation using AVX-512BW saturation arithmetic (vpaddsb, vpaddsw).