Composing Hardware-Decoded AV1 Video into the DOM

Modern web browsers offload AV1 video decoding to dedicated GPU hardware to enable power-efficient, high-resolution playback. However, integrating raw decoded video frames into the Document Object Model (DOM) alongside CSS styling, animations, and other HTML elements requires a sophisticated multi-process rendering architecture. This article details the step-by-step technical pipeline browser engines use to transition hardware-decoded AV1 video frames into the final visual compositing layer without introducing costly memory copies.

The Multi-Process Separation

Browser engines like Chromium (Blink) and Firefox (Gecko) separate media decoding, page layout, and GPU rendering into isolated processes for security and stability:

• Render Process: Executes JavaScript, builds the DOM tree, computes CSS styles, and calculates layout boundaries.
• Media / Utility Process: Demuxes the media container (e.g., WebM or MP4), extracts the AV1 bitstream, and feeds it to platform-specific APIs.
• GPU Process: Coordinates hardware access, manages GPU memory surfaces, executes WebGL/WebGPU tasks, and runs the final compositor.

Because raw 4K or 8K AV1 video consumes massive bandwidth, pixel data is never transferred back to the CPU-bound render process. Instead, rendering engines pass lightweight handles and metadata referencing GPU memory locations.

Hardware Decoding and Surface Creation

When the browser encounters an AV1 video, the media engine negotiates with the operating system’s hardware-accelerated video decoding API:

• Windows: Microsoft Direct3D 11/12 Video API (DXVA-HD / D3D11VA)
• macOS: VideoToolbox
• Linux / Android: VA-API or Android MediaCodec (using NDK AImageReader)

The hardware decoder (e.g., Intel Quick Sync, NVIDIA NVDEC, AMD VCN) writes decoded AV1 frames directly into VRAM. These are typically output in YUV color formats—specifically NV12 for 8-bit AV1 and P010 for 10-bit AV1. The output exists as an OS-level GPU surface: an ID3D11Texture2D on Windows, an IOSurface on macOS, or a dma-buf on Linux.

Representing the Video in the Compositor Tree

In the Render process, the <video> element is represented as a specialized compositor layer (such as a cc::VideoLayer in Chromium).

1. Geometry Computation: The layout engine determines the element’s screen-space transform, clip paths, opacity, and CSS filters (such as blur(), contrast(), or 3D rotations).
2. Frame Metadata Exchange: The media pipeline informs the compositor layer of new frame availability via shared memory, providing the surface identifier and presentation timestamp (PTS).
3. Layer Assignment: The compositor attaches the shared GPU surface handle to a visual quad—a textured rectangle mapped into the browser's global compositing scene graph.

Zero-Copy Texture Sharing

To composite the frame efficiently, the browser avoids memory copies by mapping the decoder's surface directly into the GPU compositor:

• Cross-API Sharing: On Windows, the decoded Direct3D 11 texture is wrapped into a shared DXGI handle, allowing the compositor to bind it as a shader resource.
• Zero-Copy Unix Primitives: On Linux, dma-buf file descriptors are imported into OpenGL/Vulkan as external textures via EGLImage or VkSamplerYcbcrConversion.
• Apple Silicon: IOSurface handles are bound directly to MTLTexture instances in the Metal-based compositor.

By relying on unified GPU memory and hardware handles, the decoded AV1 pixels never leave video memory.

The Composition Paths: Direct Overlays vs. Shader Blending

Once the GPU process receives the frame reference, it selects one of two rendering paths based on the surrounding CSS properties:

1. Hardware Overlays (The Fast Path)

If the <video> element is an unrotated rectangle with standard opacity, no overlapping DOM content, and no complex CSS filters, the browser promotes the video surface to an independent hardware overlay plane.

Using OS composition features (such as DirectComposition on Windows or Wayland subsurfaces on Linux), the browser delegates display composition directly to the GPU's display controller. The display hardware fetches the AV1 YUV surface and scans it directly onto the screen, bypassing the browser's graphics pipeline and fragment shaders entirely. This path maximizes battery life and reduces latency.

2. Shader-Based Composition (The Fallback Path)

If the video intersects with other DOM layers, uses CSS blend modes, applies CSS filters, or has complex transforms:

1. The compositor samples the decoded YUV texture inside a custom fragment shader.
2. The shader executes standard matrix multiplication to convert NV12 or P010 YUV values into linear sRGB or scRGB.
3. Tone mapping is applied if the AV1 stream includes High Dynamic Range (HDR) metadata (such as SMPTE ST 2086/BT.2020) and the output display is Standard Dynamic Range (SDR).
4. The transformed video pixels are blended into the browser’s main render target alongside the rest of the web page contents.

Synchronization and Presentation

To prevent screen tearing and frame drops, synchronization primitives (such as Direct3D fences, Metal event listeners, or EGLSyncKHR / sync points) ensure that the hardware AV1 decoder has fully completed writing a frame before the compositor or display engine attempts to sample it. Once synchronized, the composite frame is handed to the window system during the display's VSync cycle.