Immersive 360 Video Metadata Tracks in AV1

Delivering immersive 360-degree video using the AV1 video codec relies heavily on auxiliary metadata tracks embedded within the container format—predominantly ISOBMFF (MP4) aligned with MPEG-OMAF (Omnidirectional Media Format) specifications or Matroska/WebM. These auxiliary tracks provide the rendering client with precise, time-synchronized spatial context necessary to display, orient, and optimize high-resolution spherical media. This article outlines the primary auxiliary and timed metadata tracks defined for immersive AV1 video pipelines, including camera orientation, viewport guidance, region-wise packing, and depth information.

Dynamic Camera Orientation Tracks

In dynamic captures, such as those shot from moving vehicles, drones, or handheld rigs, the horizon and camera orientation frequently shift. The dynamic camera orientation track carries timed rotational data (yaw, pitch, and roll) to compensate for camera movement over time. By reading this auxiliary track, the playback client can apply rotation matrices to stabilize the sphere independently from the user’s viewport, preventing motion sickness and maintaining a stable real-world horizon.

Timed Viewport Guidance Tracks

Also known as recommended viewport or director's cut tracks, these auxiliary metadata streams define the intended focal point of a scene over time. Specified within MPEG-OMAF as timed viewport information, this metadata guides client viewports toward primary points of interest when the viewer is not actively interacting with the display, or it provides a baseline for viewport-dependent adaptive streaming optimizations.

Region-Wise Packing Metadata Tracks

High-resolution 360-degree AV1 streaming often utilizes region-wise packing, where transformed surfaces (such as cubemaps or segmented equirectangular frames) are rearranged into a single rectangular video frame to optimize compression efficiency and reduce edge distortion. When region-wise packing varies across scenes or requires dynamic tile mapping, a region-wise packing metadata track carries the spatial mapping matrices. This timed metadata informs the decoder and rendering pipeline how to extract, reposition, and stitch rectangular subsets back into their proper spherical positions.

Depth and Disparity Timed Tracks

For stereoscopic and degrees-of-freedom (DoF) extended video experiences, auxiliary depth streams can be linked to the primary AV1 video track. These tracks provide per-frame depth maps or disparity vectors encoded as monochrome video or specialized timed data. The client uses this depth metadata to generate parallax, perform reprojection for head motion compensation, and correct for stereoscopic convergence.

Spherical Projection Metadata

While often delivered as static track-level sample entry boxes (such as equirectangular projection or cubemap projection descriptors), dynamic variations in projection models can be delivered via timed metadata tracks. These tracks specify changes in projection geometry, field of view boundaries, and mesh mappings throughout playback, ensuring the rendering engine dynamically adapts its tessellation and projection shaders to match the incoming AV1 video frames.