How AV1 Compresses Stereoscopic and Multi-View 3D Video

This article provides an overview of how the AV1 video codec handles stereoscopic and multi-view 3D content. Because base AV1 does not include a dedicated multi-view profile like MVC or MV-HEVC, it compresses 3D material primarily through frame-packing formats, metadata signaling via Open Bitstream Units (OBUs), and advanced inter-frame prediction tools that treat inter-view disparity similarly to motion. Below is an explanation of the core technical mechanisms AV1 utilizes to compress and deliver 3D video content efficiently.

Frame-Packing Architectures

The primary method for stereoscopic delivery in AV1 is frame packing. Instead of encoding distinct, isolated streams for each viewpoint, both the left-eye and right-eye views are combined into a single standard 2D video frame before encoding:

• Side-by-Side (SBS): The horizontal resolution of both views is halved (or preserved in full-resolution SBS), placing the left and right perspectives next to each other within one frame.
• Top-and-Bottom (Over-Under): The vertical resolution is halved, placing one perspective on top of the other.

Because the two views share significant visual information, AV1's intra-frame and inter-frame tools compress the packed frame far more effectively than encoding two separate non-communicating streams.

Disparity Compensation via Inter-Frame Prediction

In stereoscopic video, the spatial difference between what the left and right eyes see is known as disparity. Mathematically, disparity resembles motion in standard 2D video.

When multi-view streams are structured sequentially or as alternating frames, AV1 adapts its motion estimation and compensation tools to function as disparity estimation and compensation:

• Motion Vectors as Disparity Vectors: AV1 calculates vectors pointing from a block in one view to the matching feature in the alternate view.
• Warped Motion and Affine Prediction: AV1 supports affine motion models (rotation, scaling, and shear). In multi-view video, camera convergence angles cause slight perspective distortions between views. Affine prediction allows AV1 to compensate for these angle differences, dramatically lowering residual data.

Extended Reference Frame Buffers

AV1 maintains up to eight reference frames in its decoding buffer, with the ability to reference up to seven of them for any single inter-frame prediction. In multi-view configurations:

• The codec can designate past temporal frames from the same eye's viewpoint as temporal references.
• It can simultaneously designate the current or previous frame of the opposite eye's viewpoint as an inter-view reference.
• Compound prediction combines temporal references with cross-view references, reducing bitrate requirements by reusing texture, lighting, and geometric data across perspectives.

Superblock Partitioning and Coding Efficiency

Stereoscopic images contain identical high-level scenery with minor shifts in position. AV1 utilizes 128×128 pixel superblocks that recursively split down to 4×4 blocks.

• Large Flat Areas: Backgrounds common to both views are encoded using large partitions (e.g., 64×64 or 128×128) with minimal transform coefficients.
• Detailed Edges: Near object boundaries where stereoscopic occlusions occur (areas visible to one eye but hidden from the other), AV1 breaks blocks down to finer sizes to isolate the occlusion without re-encoding shared surroundings.

Metadata Signaling with Open Bitstream Units (OBUs)

AV1 structures its bitstream into Open Bitstream Units (OBUs). To ensure decoding displays understand how to unpack 3D views, AV1 embeds layout instructions inside Metadata OBUs:

• Stereo 3D Signaling: Metadata OBUs specify whether the stream is mono, side-by-side, or top-and-bottom.
• View Assignment: Flags indicate which half of the frame corresponds to the left eye and which corresponds to the right eye, as well as whether sub-sampling was applied horizontally or vertically.
• Display Handling: Downstream display hardware parses these OBUs to slice, scale, and render the individual perspectives onto stereoscopic panels or 3D headsets without proprietary decoding layers.