How AV1 Uses Segmentation to Protect Faces

The AV1 video codec uses a powerful feature known as segmentation to dynamically adjust encoding parameters across different areas of a video frame. By pairing segmentation with Region of Interest (ROI) detection, encoders can identify visually critical areas—such as human faces—and selectively allocate more data and finer processing to them. This targeted distribution prevents noticeable compression artifacts on key subjects while keeping overall bitrates efficient by allowing background areas to carry heavier compression.

Understanding AV1 Frame Segmentation

AV1 allows an encoder to divide any given frame into up to eight distinct segments. Each segment functions as a classification layer to which a specific set of encoding rules and parameter offsets can be applied. Rather than applying uniform encoding decisions across the entire 128x128 or 64x64 superblock grid, the encoder assigns each superblock a segment ID from 0 to 7.

These segment assignments allow granular, block-level control over:

• Quantization parameter (QP) offsets
• Loop filter strengths
• Reference frame choices
• Prediction modes

Detecting the Region of Interest (ROI)

Before AV1 can protect a face, an upstream analyzer or pre-processing pipeline must identify the Region of Interest. Machine learning models, facial detection algorithms, or visual saliency maps generate bounding boxes or heatmaps around human faces in the raw video frames. The encoder then translates these spatial coordinates into a block-level map, tagging the superblocks covering facial areas with a dedicated high-priority segment ID.

Modulating Quantization (Delta QP)

The primary mechanism for preserving facial detail is the segment-level Quantization Parameter (QP) offset. Quantization controls how much high-frequency visual information is discarded to save bits:

• Lower QP for Faces: The segment assigned to the face receives a negative QP offset, lowering compression. This retains high-frequency details such as skin pores, eye clarity, hair strands, and subtle facial expressions.
• Higher QP for Backgrounds: Non-critical background segments receive a positive QP offset. Because human viewers naturally focus on human faces rather than out-of-focus or static backgrounds, higher compression can be introduced elsewhere without degrading the viewer's perceived quality.

Tuning In-Loop Filters for Fine Details

AV1 incorporates advanced in-loop filtering tools, including the Deblocking Filter, Constrained Directional Enhancement Filter (CDEF), and Loop Restoration. While these filters remove blockiness and ringing artifacts, excessive filtering can blur natural facial textures, creating an unnatural, waxy appearance.

AV1 segmentation solves this by allowing loop filter parameters to be modified per segment:

• Filter strengths can be dialed back or disabled entirely over facial segments to preserve natural sharpness and fine facial contours.
• More aggressive deblocking can remain active on flatter background segments to prevent noticeable banding or block edges.

Temporal Stability and Rate Control

Faces are particularly susceptible to temporal artifacts, such as flickering or "swimming" textures across consecutive frames. AV1 handles this by allowing temporal updates to segment maps. The encoder can preserve segment boundaries and their associated parameters across multiple frames, ensuring that facial areas maintain consistent quality over time.

By integrating segmentation directly into the encoder’s rate-control loop, AV1 guarantees that bits are actively "borrowed" from non-essential regions and reinvested into human faces, achieving a superior perceptual quality-to-bitrate ratio.