AV1 Spatial Scalability Hardware Decoder Requirements

AV1 spatial scalability enables a single video bitstream to deliver multiple resolution tiers, optimizing bandwidth for diverse playback environments without needing separate encodes. To successfully decode these multi-layered streams, hardware decoders must move beyond standard single-layer processing. This article outlines the specific architectural and functional hardware requirements—such as Operating Point parsing, Reference Picture Resampling, Decoded Picture Buffer allocation, and throughput overhead—necessary to support AV1 spatial scalability efficiently.

Operating Point (OP) Support and Bitstream Parsing

In AV1, scalable layers are organized into Operating Points (OPs). An Operating Point defines a specific combination of spatial and temporal layers. A compliant hardware decoder must:

• Parse the Sequence Header OBU: The decoder must read scalability metadata within the Sequence Header Open Bitstream Unit (OBU), identifying the available spatial layers and their dependencies.
• Filter Unselected Layers: The hardware bitstream parser must extract and discard unselected layer OBUs without CPU intervention, or process lower-dependency layers when decoding an enhancement layer that requires inter-layer prediction.

Reference Picture Resampling (RPR)

Unlike older codecs that rely on dedicated multi-view or separate scalable extensions, AV1 spatial scalability fundamentally relies on Reference Picture Resampling (RPR).

• Hardware-Based Resampling Scalers: When an enhancement layer uses a lower-resolution base layer as a reference frame, the hardware motion-compensation unit must scale the reference frame on the fly. Decoders must incorporate dedicated hardware scaling filters (typically 8-tap polyphase filters) directly inside the motion-compensation loop.
• Dynamic Resolution Switching: The hardware must switch frame buffers of differing resolutions within the same sequence without requiring a decoder re-initialization or pipeline flush.

Decoded Picture Buffer (DPB) Management

Spatial scalability increases the complexity of reference frame tracking:

• Reference Frame Slots: AV1 defines 8 reference picture slots. When decoding enhancement layers that reference reconstructed lower-layer frames, the DPB must have sufficient memory capacity and addressing logic to store both base-layer and enhancement-layer reference frames simultaneously.
• Buffer Sharing and Management: The hardware memory controller must efficiently handle the storage, eviction, and referencing of inter-layer frames according to the frame-header update rules without corrupting existing temporal reference buffers.

Throughput and Memory Bandwidth

Decoding scalable streams can introduce significant performance overhead depending on the scalability mode:

• Pixel Processing Rate: If an enhancement layer depends on base-layer reconstruction, the hardware must either decode both layers in parallel across separate cores or sequentially within a single frame interval. The hardware must have a high enough clock rate and macroblock/tile throughput to process the aggregate pixel count of all required layers.
• Memory Bandwidth Overhead: Transferring reference frames between the DPB, resampling filters, and the reconstruction pipeline requires significantly higher memory bandwidth than decoding a native single-layer stream of the same target resolution.

Profile and Level Compliance

• AV1 Main Profile Compatibility: The decoder must implement the scalability modes defined in the AV1 Main Profile (supporting 8-bit and 10-bit color in 4:2:0 chroma subsampling).
• Annex E Conformance: The hardware must conform to the decoding rules specified in Annex E of the AV1 specification, adhering to the maximum decoded sample rates and buffer capacities defined by the targeted AV1 Level when all active layers are accounted for.