Architecture of the Firefox WebRTC Media Stack

This article provides a comprehensive overview of the WebRTC media stack architecture in the Mozilla Firefox web browser, detailing how it processes real-time, peer-to-peer audio and video. It covers Firefox’s multi-process security model, the media capture and routing pipelines, the integration of the upstream libwebrtc engine for encoding and decoding, and the custom networking and cryptographic layers used to securely transport data between peers.

The Multi-Process Model

Firefox isolates the WebRTC stack across multiple processes to prevent a single failure or security vulnerability from compromising the entire browser.

  • Parent Process: Manages hardware access permissions and coordinates high-level inter-process communication (IPC).
  • Content Process: Runs the web page’s JavaScript, exposes the WebRTC APIs (like RTCPeerConnection), and manages the local media routing.
  • Socket Process: Handles direct network traffic, including STUN, TURN, and RTP packets, keeping network bottlenecks away from the user interface.
  • Gecko Media Plugin (GMP) and RDD Processes: Isolated sandboxes used for running third-party codecs (like Cisco's OpenH264) and managing hardware-accelerated decoding via the Remote Data Decoder (RDD).

Media Capture and the MediaTrackGraph

The media pipeline begins when a web application requests camera and microphone access via navigator.mediaDevices.getUserMedia().

  1. MediaManager: This component intercepts the API call, verifies user permissions through the Parent Process, and initializes the hardware drivers.
  2. Audio Capture (Cubeb): Firefox uses an internal, cross-platform library called Cubeb to interface directly with the operating system's native audio APIs (such as WASAPI on Windows, CoreAudio on macOS, and PulseAudio/PipeWire on Linux).
  3. Video Capture: Video frames are captured using OS-specific frameworks and delivered into shared memory buffers to avoid expensive data copying.
  4. MediaTrackGraph (MTG): Once captured, both audio and video streams are fed into the MediaTrackGraph. The MTG is a real-time, thread-safe processing engine that synchronizes, mixes, and routes media tracks across the browser. It acts as the central bridge between local hardware, WebAudio, <video> elements, and the WebRTC encoding engine.

Signaling and Session Management

To establish a connection, peers must exchange configuration data using the JavaScript Session Establishment Protocol (JSEP) and Session Description Protocol (SDP).

Firefox implements the RTCPeerConnection API using a C++ bridge (PeerConnectionImpl). For parsing and generating SDP offers and answers, Firefox relies on sipcc, a library originally developed by Cisco and heavily modified by Mozilla. This layer negotiates which codecs, resolutions, and encryption keys the peers will use before any media is transmitted.

Media Processing and libwebrtc Integration

Rather than building the entire real-time media engine from scratch, Firefox integrates Google’s open-source libwebrtc library for core media processing.

When the MediaTrackGraph routes raw media to an active RTCPeerConnection, the data is handed over to the libwebrtc layer. This layer is responsible for:

  • Audio Processing: Applying Acoustic Echo Cancellation (AEC),Automatic Gain Control (AGC), and noise suppression.
  • Encoding and Decoding: Compressing audio using the Opus (or G.711) codec and video using VP8, VP9, or AV1. If H.264 is negotiated, Firefox routes the frames via IPC to the sandboxed GMP process.
  • Packetization and Jitter Management: Slicing encoded frames into Real-time Transport Protocol (RTP) packets for transmission, and reassembling incoming packets in a jitter buffer to smooth out network irregularities.
  • Bandwidth Estimation: Processing Real-time Transport Control Protocol (RTCP) feedback to dynamically adjust video bitrates and frame rates based on network congestion.

Networking, Security, and Transport

While Firefox uses libwebrtc for media processing, it replaces libwebrtc's default networking and cryptography modules with Mozilla’s own battle-tested components. This happens primarily within the dedicated Socket Process.

  • nICEr: To traverse NATs and firewalls, Firefox uses an internal C-based library called nICEr to implement Interactive Connectivity Establishment (ICE), STUN, and TURN protocols.
  • NSS (Network Security Services): WebRTC mandates encryption for all media. Firefox uses its native NSS library to perform the DTLS (Datagram Transport Layer Security) handshake.
  • SRTP: Once the DTLS handshake is complete, the negotiated keys are used to encrypt the media payloads using Secure RTP (SRTP) before they are sent over UDP or TCP sockets to the remote peer.

Playback and Rendering

When encrypted media packets arrive from a remote peer, the architecture executes the pipeline in reverse:

  1. The Socket Process receives the packets via nICEr, decrypts the SRTP payload, and forwards the RTP packets to the Content Process.
  2. The libwebrtc engine places the packets into a jitter buffer, reconstructs the frames, and decodes them (utilizing the RDD or GMP process if hardware acceleration or H.264 is used).
  3. Decoded frames are injected back into the MediaTrackGraph.
  4. Audio is pushed from the MTG to Cubeb for immediate playback through the user's speakers.
  5. Video frames are sent from the MTG directly to WebRender (Firefox's GPU-based compositor), which renders the video onto the screen within the designated HTML <video> element.