Encapsulating PCM and Opus Audio with AV1

This article explains how uncompressed PCM and high-resolution Opus audio streams are paired and encapsulated alongside AV1 video. Because AV1 is strictly a video coding format rather than a multimedia container, it does not package audio on its own. Instead, it relies on standardized container formats—primarily the ISO Base Media File Format (MP4) and Matroska (MKV/WebM)—to interleave, synchronize, and transport AV1 video tracks with audio data.

The Role of AV1 and Media Containers

AOMedia Video 1 (AV1) defines only the bitstream syntax and decoding process for video frames. Audio codecs like Opus and uncompressed Linear Pulse-Code Modulation (LPCM) operate entirely independently of the AV1 compression engine. To play audio and AV1 video together, multiplexers wrap the individual elementary streams into container tracks using specific track headers, sample descriptions, and timestamps.

Encapsulating Opus with AV1

Opus is a lossy, low-latency audio format capable of handling high-resolution signals up to a 48 kHz sampling rate with dynamic bitrates. Encapsulating Opus alongside AV1 varies by container:

• ISO Base Media File Format (ISOBMFF / MP4): Under standard specifications governed by the Alliance for Open Media (AOMedia), an Opus track inside an MP4 file uses the sample entry four-character code 'Opus'. The track includes an Opus Specific Box (dOps) containing essential initialization data: the version, channel count, pre-skip (codec delay samples that must be discarded upon decoding), original sample rate, and output channel mapping. Individual Opus packets are stored as discrete samples mapped to presentation timestamps (PTS) matching the AV1 composition times.
• Matroska (MKV) and WebM: Matroska-based containers identify Opus tracks via the CodecID set to A_OPUS. The track's CodecPrivate element holds the Opus identification header, detailing channel configuration and pre-skip values. Opus packets are written directly into Matroska SimpleBlock or Block structures, interleaved chronologically with AV1 video blocks to maintain low buffer requirements during streaming and playback.

Encapsulating Uncompressed PCM with AV1

PCM audio contains raw, uncompressed sample values that require precise format definitions regarding endianness, bit depth, and layout.

• ISO Base Media File Format (ISOBMFF / MP4): PCM audio encapsulation in MP4 relies on specific audio sample entries. Historical QuickTime-derived implementations use 'sowt' (little-endian signed integer) or 'twos' (big-endian signed integer). The modern standardized approach uses the 'lpcm' or 'in24'/'in32' boxes, which declare parameters such as sample size (e.g., 16-bit, 24-bit, 32-bit), channel count, and sampling rate (e.g., 96 kHz, 192 kHz). PCM samples are packed into chunks with fixed durations, allowing decoders to linearly map audio samples to AV1 video frames.
• Matroska (MKV): Matroska explicitly supports uncompressed audio via defined CodecID strings:
  • A_PCM/INT/LIT for signed, little-endian integer data.
  • A_PCM/INT/BIG for big-endian integer data.
  • A_PCM/FLOAT/IEEE for 32-bit or 64-bit floating-point audio. Track settings define the bit depth and sampling frequency. While Matroska fully supports high-resolution multichannel PCM combined with AV1, the subset format WebM excludes raw PCM, restricting audio tracks strictly to Opus and Vorbis.

Synchronization and Multiplexing

Synchronization between AV1 video and the chosen audio stream is maintained at the container level through common timebases and presentation timestamps:

• Timestamp Alignment: Containers assign timestamps relative to a global clock (such as the Matroska TimecodeScale or MP4 timescale). Demuxers use these values to feed the AV1 decoder and the audio decoder simultaneously, compensating for playback drift.
• Preroll and Delay Compensation: High-resolution Opus relies on encoder delay and pre-skip data to achieve seamless frame transitions. The container communicates this preroll data to ensure the audio timeline begins at the exact microsecond the first AV1 video frame displays.
• Interleaving Strategy: Multiplexers interleave audio and video chunks at small intervals (typically 0.5 to 1 second of media per chunk). This guarantees that client players buffer raw PCM or Opus data in parallel with AV1 temporal units without excessive memory overhead.