AV1 Timecode and Synchronization Structures Explained

This article explores how the AOMedia Video 1 (AV1) format handles timing, synchronization, and frame scheduling across both its elementary bitstream and standard container formats like ISOBMFF (MP4) and Matroska (MKV/WebM). While the AV1 bitstream provides foundational timing information and SMPTE timecode metadata, enclosing container formats manage presentation timestamps, decode timestamps, sample dependencies, and cross-track synchronization to ensure accurate media playback.

Bitstream-Level Timing in AV1

AV1 decouples internal display timing from the transport layer, but it embeds specific structures within Open Bitstream Units (OBUs) to define intrinsic frame rates and real-world timecodes.

1. Sequence Header Timing Information

Within the Sequence Header OBU, AV1 provides optional native timing mechanisms via the timing_info structure:

• timing_info_present_flag: Indicates whether bitstream timing parameters are signaled.
• num_units_in_display_tick and time_scale: Define the tick frequency (similar to a clock rate), establishing the time interval between display ticks.
• equal_picture_interval: Signaled to indicate whether every frame is displayed for an identical duration (constant frame rate).

2. Metadata OBU (Timecode)

AV1 defines a dedicated Metadata OBU with a type set to METADATA_TYPE_TIMECODE. This structure embeds traditional broadcast-style timecodes directly into the video stream:

• Adheres to SMPTE 12-1 / ISO/IEC 15444-1 conventions.
• Fields include hours, minutes, seconds, frames, and flags such as drop_frame_flag.
• Useful for linear video editing, broadcast environments, and frame-accurate archival where container timestamps might be stripped or rewritten.

Synchronization Structures in ISOBMFF (MP4) Containers

The ISO Base Media File Format (defined for AV1 under the Alliance for Open Media AV1-ISOBMFF specification) is the standard container for web streaming (DASH, HLS) and offline MP4 playback. It manages timecodes and frame ordering through sample table boxes.

Presentation and Decode Timestamps

AV1 frames support out-of-order decoding (display ordering differs from decode ordering due to inter-frame dependencies).

• stts (Time-to-Sample Box): Stores Decode Timestamps (DTS), mapping each sample to its decode interval.
• ctts (Composition Time-to-Sample Box): Provides the composition offset needed to calculate the Presentation Timestamp (PTS) from the DTS:
  \[\text{PTS} = \text{DTS} + \text{Composition Offset}\]
• tfdt (Track Fragment Base Media Decode Time Box): In fragmented MP4 (fMP4) streams, this box encodes absolute decode timestamps for the first sample of a media fragment, ensuring continuous synchronization across live streaming boundaries.

Keyframe and Random Access Structures

• stss (Sync Sample Box): Catalogs random access points (Key Frames / Intra Frames). An AV1 Key Frame or a Key Frame with delayed random access (Switch Frames or Intra-only Frames preceded by an S-Frame) is flagged here.
• sdtp (Sample Dependency Table Box): Explicitly signals whether a sample depends on others, if other samples depend on it, or if it can be safely discarded without breaking the decoding chain.

Cross-Track Audio/Video Synchronization

• elst (Edit List Box): Maps presentation times on the media timeline to the master movie timeline. Edit lists resolve track offsets, delay video starts, and handle non-zero initial presentation times to keep audio and video tracks aligned.

Synchronization Structures in Matroska (MKV) and WebM Containers

Matroska and WebM encapsulate AV1 OBUs within hierarchical binary elements (EBML). Timing and synchronization operate at the Block level rather than through centralized offset tables.

Timestamp Tracking

• TimestampScale: A segment-level header element defining the duration multiplier (default is 1,000,000 nanoseconds, or 1 millisecond).
• Timestamp: Embedded within each Cluster header, establishing the base presentation time for that cluster.
• Block / SimpleBlock Timestamps: Individual AV1 frames store a signed 16-bit relative offset added to the cluster's base Timestamp. In WebM and Matroska, timestamps directly represent presentation times (PTS).

Decoding Dependencies and Seeking

• SimpleBlock Flags: Contains a 1-bit Keyframe flag signaling if the encapsulated AV1 frame is an independent synchronization point.
• ReferenceBlock: Used in standard BlockGroup structures to explicitly declare the relative timestamp of referenced forward or backward anchor frames.
• Cues Element: Contains CuePoint, CueTime, and CueTrackPositions. This acts as an index for seek operations, pointing the demuxer to the exact byte offset of keyframes matching a given presentation timecode.

Summary of Interaction