Dynamic Screen Resizing in AV1 Scalable Streams

This article explains how the AV1 video codec manages dynamic screen resizing within spatially scalable streams. It details the interplay between AV1's Reference Frame Resampling (RFR), spatial Scalable Video Coding (SVC) modes, and normative downsampling filters. By examining these mechanisms, you will learn how AV1 dynamically changes spatial resolutions without relying on expensive keyframes, maintaining uninterrupted playback and bandwidth efficiency during viewport alterations.

The Role of Reference Frame Resampling (RFR)

Traditional video codecs typically require an Instantaneous Decoder Refresh (IDR) or keyframe to change stream resolution, which causes significant bitrate spikes and decoder latency. AV1 solves this limitation natively through Reference Frame Resampling (RFR).

In an AV1 stream, inter-prediction frames can reference previous frames that were encoded at different spatial resolutions. When dynamic screen resizing occurs—such as a user dragging a browser window or switching between full-screen and tiled layouts—the encoder can alter the spatial downsampling ratio on any inter-frame. The AV1 decoder dynamically scales reference frames in memory to match the target frame's dimensions using high-precision 8-tap interpolation filters before executing motion compensation.

Spatial Downsampling within SVC Frameworks

AV1 integrates spatial scalability natively into its bitstream structure through operating points and scalability structures defined in the sequence header. Spatial scalability relies on splitting the video feed into a base layer and one or more spatial enhancement layers:

• Base Layer: Encoded at a downsampled resolution (e.g., 50% or 25% of the original capture).
• Enhancement Layers: Encoded at higher resolutions, utilizing inter-layer prediction from the lower spatial layers as well as temporal prediction from prior frames.

When a client dynamically resizes its display area, the receiving device or an intermediate Selective Forwarding Unit (SFU) responds by altering which spatial layer is actively forwarded or decoded. If a user shrinks the playback window, the delivery pipeline can drop higher spatial enhancement layers entirely, routing only the spatially downsampled base layer. Because the layers are temporally synchronized, this transition happens seamlessly without renegotiating the codec session.

Frame Header Signaling and Coordinate Mapping

Spatial downsampling adjustments during dynamic resizing are governed frame-by-frame via the frame_size_override_flag and render_and_frame_size_info syntax elements in the AV1 uncompressed header.

When a screen resize event triggers a resolution shift:

1. Dimension Signaling: The frame header specifies the coded width and height alongside optional render width and height attributes.
2. Motion Vector Scaling: Motion vectors pointing to reference frames with differing dimensions are automatically scaled mathematically. AV1 maps the coordinates using fractional scaling factors based on the ratio between the current and reference frame dimensions.
3. Filter Application: The standard AV1 normative downscaler and upscaler handle fractional phase offsets, preventing drift between the reference buffers and newly reconstructed surfaces.

Native Super-Resolution as a Downsampling Tool

In addition to traditional spatial SVC layer configurations, AV1 includes an integrated in-loop Super-Resolution tool. Super-resolution functions by applying a horizontal-only spatial downsampling step to the source frame prior to encoding, processing the frame at lower complexity, and applying a normative upscaling filter immediately after the loop restoration stage inside the decoding loop.

During rapid dynamic screen resizing, an encoder can engage horizontal super-resolution dynamically. This allows the system to lower effective bitrate and processing overhead instantly without restructuring the global reference buffer dimensions, acting as a lightweight spatial resizing tool that stabilizes playback until the display dimensions settle.