AV1 Temporal Scalability in Bitstream Framing

This article provides an overview of how the AV1 video codec natively supports temporal scalability through its bitstream syntax. By leveraging Open Bitstream Units (OBUs), hierarchical reference structures, and explicit header signaling, AV1 enables decoders or media servers to drop higher temporal layers to adjust frame rates dynamically without requiring full bitstream re-encoding or corrupting baseline playback.

Open Bitstream Units and the Extension Header

AV1 encapsulates all video data—including sequence metadata, frame headers, and compressed pixel data—into modular containers called Open Bitstream Units (OBUs). Temporal scalability is signaled directly at the framing level using the obu_extension_header.

When scalability is enabled via the sequence header, OBUs carry an optional 1-byte extension header containing two vital syntax elements:

• temporal_id (3 bits): Explicitly designates the temporal layer (from Layer 0 up to Layer 7) to which the OBU belongs.
• spatial_id (2 bits): Designates spatial layers for spatial scalability, allowing temporal and spatial layers to coexist.

A network middlebox or selective forwarding unit (SFU) can read the temporal_id directly from the OBU extension header without parsing deep into the uncompressed frame header or decoding the payload, allowing it to discard higher-tier packets on congested links seamlessly.

Temporal Units and Delimiters

AV1 bitstreams are structured into Temporal Units (TUs). Each TU represents a collection of OBUs that share the same presentation timestamp.

A Temporal Unit begins with a dedicated OBU_TEMPORAL_DELIMITER. This zero-payload unit marks the boundary between consecutive temporal instances in the bitstream. Within a single TU, there can be multiple frames or metadata elements corresponding to different temporal or spatial layers. By pairing delimiters with the temporal_id, decoders can unambiguously isolate all frames associated with a given time tick.

Operating Points and Bitstream Extraction

AV1 formally defines scalable bitstream subsets as Operating Points. In the Sequence Header OBU, the bitstream declares the number of operating points and their capabilities via the operating_point_idc parameter:

• operating_point_idc (12 bits): A bitmask that specifies which combinations of spatial and temporal layers form a compliant, independently decodable stream.

The first 8 bits of operating_point_idc directly map to temporal_id inclusion. If an operating point targets temporal layer 2, it instructs the decoder to parse all OBUs where temporal_id <= 2 while dropping all higher IDs.

Reference Frame Management and Prediction Rules

Temporal scalability relies on a non-destructive reference buffer strategy. AV1 maintains eight reference frame buffer slots, and its framing ensures lower temporal layers never depend on higher ones:

1. Prediction Constraints: Frames with a given temporal_id are strictly restricted to predict from frames with an equal or lower temporal_id.
2. Buffer Refresh Controls: The frame header contains a refresh_frame_flags syntax element (an 8-bit mask). Frames in higher temporal layers (such as sub-layers used strictly for high frame rates) set their refresh flags to zero or update only auxiliary buffers, preventing them from overwriting the base reference frames needed by temporal_id = 0.
3. AltRef Frames: AV1 utilizes non-displayed alternative reference frames (AltRefs). These are encapsulated in distinct OBUs within the bitstream, allowing complex hierarchical prediction trees (such as dyadic structures like 30 fps to 60 fps) to be signaled cleanly within the framing standard.