How AV1 In-Loop Deblocking Filters Remove Artifacts

This article provides an overview of the in-loop deblocking filter in the AV1 video codec, explaining how it identifies and mitigates artificial block edges caused by transform and prediction quantization. It covers the root causes of boundary discontinuities, the mechanics of loop filtering within the decode pipeline, adaptive edge selection, and the mathematical smoothing operations used to preserve authentic image details while restoring visual continuity.

The Origin of Block Boundary Artifacts

Modern video codecs like AV1 rely on block-based hybrid coding. Images are partitioned into rectangular blocks known as Coding Block Units (ranging from 4x4 to 128x128 pixels) to perform spatial prediction, temporal motion compensation, and frequency transforms. When these blocks undergo quantization—the lossy step that compresses the data—high-frequency transform coefficients are discarded.

Because neighboring blocks are predicted and quantized independently, the reconstruction errors at the edges do not match. This mismatch produces sharp, unnatural step discontinuities along the boundaries of transform blocks and prediction blocks. To human vision, these manifest as a distracting grid-like pattern across continuous surfaces such as skies, walls, and faces.

The Role of an "In-Loop" Design

AV1 implements its deblocking filter "in-loop," which distinguishes it from post-processing filters:

1. Reference Frame Consistency: The deblocking filter processes the reconstructed frame before it is stored in the Decoded Picture Buffer (DPB).
2. Error Drift Prevention: Subsequent frames that use inter-picture prediction reference the deblocked frame rather than the unfiltered, raw reconstructed frame. This prevents boundary errors from compounding and propagating across multiple frames over time.

Edge Identification and Processing Order

The AV1 deblocking filter operates on a virtual 4x4 sample grid across the entire frame. It processes two primary edge types:

• Vertical Boundaries: Filtered first to modify horizontal pixel values across vertical block seams.
• Horizontal Boundaries: Filtered second to modify vertical pixel values across horizontal block seams.

The filter only targets edges corresponding to actual transform block boundaries or prediction block boundaries. Interior regions of a transform block are skipped because they do not contain quantization-induced boundary steps.

Adaptive Filter Strength and Thresholding

To avoid blurring real image features like sharp object outlines or textures, the filter dynamically adjusts its intensity based on several parameters:

• Quantization Parameter (QP): Higher quantization introduces more severe artifacts, prompting the filter to apply higher base filtering strengths. Lower QP values reduce filter strength to preserve fine texture.
• Prediction Modes: Intra-coded blocks usually have higher reconstruction variances at their edges than inter-coded blocks, often warranting stronger filtering.
• Motion Vectors: If neighboring inter-blocks share similar motion vectors and the same reference frame, the likelihood of a severe prediction mismatch is reduced, resulting in weaker filtering.
• Frame and Block-Level Deltas: AV1 allows the encoder to signal global filter strength levels per frame, with optional segment-level or block-level adjustments to handle local complexity.

Boundary Condition Testing

Before any pixel values are modified, the filter tests the boundary samples to ensure that the discontinuity is an artifact rather than a true edge:

1. Samples perpendicular to the boundary (designated as \(p_2, p_1, p_0\) on one side and \(q_0, q_1, q_2\) on the other) are sampled.
2. The filter evaluates flatness conditions, checking whether \(|p_0 - q_0|\), \(|p_1 - p_0|\), and \(|q_1 - q_0|\) stay below defined threshold values derived from the filter strength parameters.
3. If the sample differences exceed the threshold, the algorithm classifies the transition as an authentic image edge, and filtering is suppressed to preserve sharpness.

Narrow and Wide Filter Applications

Once an edge passes the boundary condition tests, AV1 applies either a narrow or a wide filter depending on the block size and sample flatness:

• Narrow Filters (2-tap to 4-tap): Applied to smaller transform blocks (such as 4x4) or boundaries with moderate gradient variations. The filter alters only the immediate boundary samples (\(p_0\) and \(q_0\), and sometimes \(p_1\) and \(q_1\)) by applying a clipping-limited delta to smooth out the step discontinuity without affecting surrounding pixels.
• Wide Filters (up to 14-tap): Applied to larger transform boundaries (such as 8x8, 16x16, 32x32, or 64x64 blocks) where severe blocking artifacts can create large-scale banding across flat areas. Wide filters sample deeper into the block (up to 7 pixels on each side) and apply multi-tap weighted averaging. This distributes the transition smoothly over a wider area without introducing noticeable blurring.

Independent Color Channel Processing

AV1 applies deblocking to luma (\(Y\)) and chroma (\(Cb, Cr\)) planes independently. Because the human visual system perceives brightness contrasts and color transitions differently, luma and chroma require distinct thresholding. The encoder can specify separate filter levels for vertical and horizontal edges across both luma and chroma components, preventing color bleeding across boundaries while maintaining edge cleanliness.