AV1 Spatial Scalability in Video Conferencing

This article explores how the AV1 video codec implements spatial Scalable Video Coding (SVC) to optimize multi-party video conferencing. By generating a single bitstream that contains multiple resolution layers, AV1 enables real-time communication systems to dynamically serve participants with differing screen sizes and network conditions without placing heavy computational burdens on client encoders or conference servers.

Native Bitstream Support for Scalability

Unlike legacy standards such as H.264, which required a complex and poorly adopted extension (SVC) to achieve layered coding, AV1 incorporates scalable video coding directly into its core specification. AV1 achieves this through Open Bitstream Units (OBUs) and defined "Operating Points."

An AV1 encoder includes scalability structure metadata within sequence headers. This metadata explicitly defines the spatial layers (e.g., 720p, 360p, and 180p) packaged within the stream. Decoders and media servers can parse this metadata to understand layer dependencies without needing to decode the underlying video frames.

Inter-Layer Reference and Coding Efficiency

AV1 spatial scalability uses inter-layer prediction to maximize compression efficiency across different resolutions. In standard simulcast, a client encodes multiple independent video streams, duplicating motion estimation and picture details across every resolution.

With AV1 spatial SVC, the encoder uses its flexible reference frame buffer—which allows tracking up to eight reference frames simultaneously—to share data between layers:

• Base Layer: The lowest spatial resolution (e.g., 180p or 360p) is encoded independently using standard temporal references.
• Enhancement Layers: Higher-resolution layers (e.g., 720p or 1080p) can reference both their own previous temporal frames and the upscaled reconstructed frames of the base layer within the same time instance.

This inter-layer prediction significantly reduces the bitrate overhead compared to simulcasting multiple discrete video feeds.

Selective Forwarding Unit (SFU) Routing

In multi-party conference calls, media servers typically operate as Selective Forwarding Units (SFUs). An SFU does not decode or transcode incoming video; instead, it selectively routes video packets to participants based on their downstream network conditions and layout requirements.

AV1 spatial scalability integrates directly into this architecture:

1. Layer Dropping: If a participant experiences packet loss or bandwidth constraints, the SFU can strip away the higher-resolution enhancement layer packets and forward only the lower-resolution base layer.
2. Dynamic Resolution Switching: When a conference participant changes layout—such as switching a speaker from a small thumbnail to a full-screen view—the SFU immediately begins forwarding higher enhancement layers to that participant without requesting an IDR keyframe from the sender.
3. Low Latency: Because switching occurs by simply filtering packets at the network layer, there is zero transcoding delay introduced by the server.

Client-Side Resource Optimization

Spatial SVC in AV1 reduces local device workload compared to simulcast. Instead of running two or three separate hardware encoder pipelines simultaneously, the device runs a single scalable encoder pipeline. This lowers memory bandwidth usage, reduces power consumption on mobile and laptop hardware, and maintains stable framerates during intensive multi-party calls.