Storing Portrait Mode Depth Data in AVIF

Computational photography systems rely on depth maps to simulate optical shallow depth of field, adjust focal points after capture, and apply relighting effects. The AVIF (AV1 Image File Format) specification provides a standardized, highly compressed container to store this depth data alongside the primary color image. This article explains how computational photography pipelines generate depth maps, how the underlying container architecture links auxiliary items to primary imagery, and how AV1 compression encodes spatial depth for post-capture processing.

The ISOBMFF Container and Auxiliary Images

AVIF is derived from the High Efficiency Image File Format (HEIF), which is built on top of the ISO Base Media File Format (ISOBMFF). Rather than flattening depth information into custom, proprietary metadata fields—such as base64-encoded JPEGs embedded in XMP tags—AVIF natively supports multi-item architectures.

Within an AVIF file, the primary image and the depth map exist as distinct image items stored in the same file container. The primary color image (typically a full-color YUV 4:2:0 or 4:4:4 image) is designated as the primary item (pitm). The depth map is saved as a separate item marked as an auxiliary image item.

The relationship between these two items is established using an item reference box (iref) with a reference type of auxl (auxiliary). This reference explicitly links the depth data item to the primary image item, signaling to decoders that the secondary item is not intended for standalone display, but instead modifies or supplements the primary image.

Auxiliary Type Identification

To allow image processing software to interpret the auxiliary data correctly, the file includes an Auxiliary Type Property box (auxC). This property contains a Uniform Resource Name (URN) that defines the exact nature of the data.

For depth maps, standard identifier strings indicate whether the data represents metric depth (absolute distance in meters) or disparity (inverse distance, common in stereo-camera systems). Standard identifiers include standardized MPEG depth URNs (such as urn:mpeg:heif:2015:auxid:depth) or platform-specific definitions. Decoders read this URN to determine whether values need inversion, scaling, or special handling to accurately recreate the virtual camera's focal plane.

Monochrome AV1 Compression for Depth Maps

Depth data generated by stereo vision, dual-pixel sensors, or LiDAR sensors is fundamentally a single-channel 2D spatial array. AVIF leverages the AV1 video codec's native monochrome profile to encode this array with high efficiency:

• Single-Channel Encoding: The depth map is encoded purely as a luma (Y) channel, discarding unnecessary chroma planes to save space.
• Bit Depth: While standard color images are often 8-bit, depth maps often benefit from 10-bit or 12-bit monochrome AV1 encoding. This higher bit depth prevents contouring and banding in smooth gradients, which is critical for realistic bokeh transitions.
• Lossy vs. Lossless Compression: Algorithms usually compress depth maps using lossy AV1 encoding tuned to preserve sharp object boundaries (edges) while aggressively compressing smooth background regions where precision loss does not noticeably impact blur quality.
• Resolution Independence: The depth item does not need to match the resolution of the primary image. Computational algorithms often downsample the depth map to half or quarter resolution to minimize file size, relying on post-processing edge-aware upsampling (such as bilateral filtering) to match the primary image at render time.

Storing Calibration and Range Metadata

Raw depth pixels represent values within a normalized range (typically 0 to 255 for 8-bit, or 0 to 1023 for 10-bit). To convert these integer values into physical distances for bokeh calculation, the AVIF file includes auxiliary metadata embedded in EXIF, XMP, or custom item properties:

1. Near and Far Planes: Metadata specifies the physical distance corresponding to the minimum (0) and maximum pixel values, allowing downstream software to map values to real-world meters.
2. Disparity vs. Depth Flags: The metadata clarifies whether a higher value corresponds to a point closer to the camera (disparity) or farther away (linear depth).
3. Camera Intrinsics: Parameters like focal length, principal point, and lens distortion coefficients can be embedded to allow synthetic aperture rendering that accurately reflects realistic optical flaws.

Backward Compatibility and Rendering

When an application without computational photography capabilities opens an AVIF file, the decoder queries the pitm box and extracts only the primary color image, ignoring auxiliary items. Systems equipped with portrait-processing engines detect the auxl relationship, decode the monochrome AV1 stream into memory as a separate texture buffer, and feed both the primary image and the depth buffer into shaders responsible for depth-of-field blur, synthetic lighting, or 3D parallax effects.