Maximum Pyramid Depth for AV1 AltRef Frames

This article provides an overview of Alternative Reference (AltRef) frames in the AV1 video codec and details their hierarchical organization. It specifically examines the maximum pyramid depth commonly utilized in standard AV1 implementations, explaining how this depth influences compression efficiency, memory consumption, and encoding latency.

What Are AltRef Frames in AV1?

In the AV1 video coding standard, Alternative Reference (AltRef) frames are synthetic, non-displayable frames generated by the encoder. Rather than being shown directly to the viewer, an AltRef frame is constructed using temporal filtering (averaging multiple frames across time) to remove noise and serve purely as a high-quality predictor for other frames.

To maximize coding efficiency, encoders organize AltRef frames and standard inter-frames into a hierarchical or "pyramid" structure, commonly referred to as a mini-GOP (Group of Pictures).

The Maximum Commonly Used Pyramid Depth

In standard AV1 implementations—such as the reference encoder libaom and the production-oriented SVT-AV1—the maximum pyramid depth commonly used for AltRef structures is 5.

• Depth 4 (Mini-GOP of 16): This is the most widespread setting across standard encoding presets. It provides a strong balance between high compression efficiency, reasonable encoding speed, and manageable memory overhead.
• Depth 5 (Mini-GOP of 32): This represents the practical upper ceiling commonly supported and deployed in mainstream AV1 encoders. It introduces a 5-level hierarchical prediction tree with multi-layer AltRef frames.

While theoretical architectures can extend further, depths beyond 5 are rarely implemented in practice due to steep diminishing returns in coding gain coupled with substantial resource penalties.

How Pyramid Depth Operates

In a hierarchical AltRef structure, frames are assigned to different temporal layers:

1. Base Layer (Depth 0): Keyframes and primary anchor frames that define the boundaries of the mini-GOP.
2. Intermediate AltRef Layers (Depths 1 to 4): Lower-level AltRef frames positioned at temporal midpoints. Each subsequent layer references the filtered AltRef frames above it.
3. Leaf Layer (Depth 5 in a 32-frame mini-GOP): Standard display frames that use the surrounding multi-layer AltRef frames for bidirectional prediction.

Trade-offs of Deeper AltRef Pyramids

Increasing the pyramid depth from 3 or 4 up to 5 affects the encoding process in several ways:

• Coding Efficiency: Deeper hierarchies improve temporal prediction, lowering the average bitrate required for complex motion and noisy scenes.
• Lookahead and Latency: A pyramid depth of 5 requires a lookahead buffer of at least 32 frames to calculate temporal filters and multi-pass statistics, making it unsuitable for real-time or low-latency streaming.
• Memory Usage: The decoder and encoder must retain more reference frames in the DPB (Decoded Picture Buffer). AV1 allows up to 8 reference frames to be tracked simultaneously, which naturally bounds the practical complexity of deeper pyramids.