How WebRTC Natively Ensures Audio Video Lip-Sync

WebRTC delivers seamless real-time communication, but keeping separate audio and video tracks in perfect synchronization—commonly known as lip-sync—presents a significant technical challenge. This article explains how WebRTC natively achieves this alignment by mapping independent Real-Time Transport Protocol (RTP) streams to a common wall-clock time using RTCP Sender Reports, allowing the receiver to reconstruct a perfectly synchronized playback.

The Challenge of Separate Streams

In WebRTC, audio and video are captured by different hardware devices, encoded separately, and transmitted over the network as completely independent RTP streams. Because they use different sampling rates—typically 48 kHz for audio and 90 kHz for video—their RTP packet timestamps use different scales and start at different random offsets. Consequently, a receiver cannot synchronize the streams by simply comparing their raw RTP timestamps.

The Role of RTCP Sender Reports

To bridge the gap between the different timelines of the audio and video streams, WebRTC relies on the RTP Control Protocol (RTCP). Periodically, the sender transmits RTCP Sender Reports (SR) for each active media track.

Each RTCP Sender Report contains a crucial pairing of two timestamps: 1. The NTP Timestamp: The absolute wall-clock time of the sender’s system (based on Network Time Protocol) at the moment the packet was sent. 2. The RTP Timestamp: The corresponding RTP media timestamp of the stream at that exact same moment.

By sending this paired data, the sender provides the receiver with a translation key for each individual stream.

Synchronization at the Receiver

When the receiver receives the audio and video packets alongside their respective RTCP Sender Reports, it performs the following steps to ensure lip-sync:

  1. Establish a Common Timeline: The receiver uses the NTP-to-RTP mapping from the RTCP Sender Reports to calculate the absolute wall-clock time for every incoming audio and video packet. This maps both streams onto a single, unified timeline.
  2. Calculate Relative Drift: The receiver monitors the arrival of both streams. If one stream begins to lag behind the other due to network jitter, processing delays, or thread scheduling, the receiver detects the discrepancy using the common NTP timeline.
  3. Adjust Playback Buffers: To align the streams, the WebRTC receiver’s jitter buffer introduces tiny, imperceptible delays. Usually, because video processing takes longer than audio decoding, the receiver will slightly delay the audio playback to match the rendering time of the corresponding video frame.

Stream Grouping via SDP

For this synchronization to occur, the receiver must know which audio and video tracks belong together. WebRTC achieves this during the signaling phase using the Session Description Protocol (SDP).

Tracks that originate from the same media source are grouped together using the msid (Media Stream ID) attribute. When the receiver identifies an audio track and a video track sharing the same Media Stream ID, it automatically binds them together in its synchronization engine, triggering the RTCP-based lip-sync mechanism.