AV1 Temporal Scalability and Frame Rate Drops

The AOMedia Video 1 (AV1) codec employs a hierarchical temporal layer structure within its scalable video coding framework to facilitate smooth, artifact-free frame-rate drops. By organizing video frames into nested dependency tiers identified by temporal IDs, AV1 allows streaming servers and decoders to selectively discard higher-layer frames during network congestion or processing bottlenecks. This architecture ensures that dropping frames reduces the playback frame rate cleanly without disrupting the decoding reference chain or causing visual corruption.

Hierarchical Prediction Structure

AV1 achieves temporal scalability by structuring frames in a dyadic, tree-like hierarchy. In this structure, frames are categorized into a base temporal layer (\(T_0\)) and one or more enhancement layers (\(T_1, T_2, \dots, T_n\)):

• Layer 0 (\(T_0\) - Base Layer): Represents the lowest frame-rate tier (e.g., 15 frames per second in a 60 fps stream). These frames are essential and only reference other \(T_0\) frames.
• Layer 1 (\(T_1\) - First Enhancement Layer): Interleaved temporally between \(T_0\) frames. When combined with \(T_0\), it doubles the effective frame rate (e.g., from 15 to 30 fps). \(T_1\) frames can reference \(T_0\) frames or preceding \(T_1\) frames, but never higher layers.
• Layer 2 (\(T_2\) - Second Enhancement Layer): Interleaved between \(T_0\) and \(T_1\) frames to bring the stream up to full frame rate (e.g., 60 fps). Frames in \(T_2\) can reference both \(T_0\) and \(T_1\) frames.

Reference Picture Constraints

The defining rule of AV1's temporal layer structure is strictly unidirectional dependency: a frame at temporal layer \(T_k\) can only use reference pictures from layers where \(T \le k\). It is strictly prohibited from referencing any picture belonging to a layer higher than itself (\(T > k\)).

AV1 supports up to eight reference frame buffers, providing the encoder with the flexibility to maintain long-term base-layer references while simultaneously referencing intermediate enhancement frames. Because higher-layer frames are never used as prediction references for lower-layer frames, discarding an entire enhancement layer leaves the lower-layer dependency chain completely intact.

Temporal Signaling in the Bitstream

AV1 signals temporal layers at the transport and container level via Open Bitstream Units (OBUs). Each OBU can include an extension header that explicitly defines:

• temporal_id: Identifies the exact temporal tier (0 through 7) to which the OBU belongs.
• spatial_id: Identifies spatial resolution layers, allowing temporal scaling to operate alongside spatial scalability.

Because the temporal_id is exposed in the OBU header, network entities such as Selective Forwarding Units (SFUs) in WebRTC do not need to parse or transcode the raw video payload. The routing infrastructure reads the header and drops high-index temporal OBUs on the fly.

Mechanism for Smooth Frame-Rate Drops

When a client experiences bandwidth degradation or CPU saturation, the sender or intermediary server can drop enhancement layers progressively:

1. Full Quality (e.g., 60 fps): Transmits \(T_0 + T_1 + T_2\).
2. First Drop (e.g., 30 fps): Discards all \(T_2\) frames. Playback remains visually stable and continuous at half frame rate without macroblocking or missing reference errors.
3. Second Drop (e.g., 15 fps): Discards both \(T_2\) and \(T_1\) frames, leaving only the \(T_0\) base stream.

Because the temporal spacing between frames within any given sub-layer is uniform, dropping layers results in an evenly spaced cadence of delivered frames rather than irregular, stuttering drops. Once operating conditions improve, transmission of higher temporal layers can resume instantly at the next compatible temporal boundary without waiting for a keyframe.