AV1 Reference Frame Resampling Explained

The Reference Frame Resampling (RFR) tool in the AV1 video codec allows an encoder to dynamically change video resolution on a per-frame basis without requiring a keyframe. By enabling inter-frames to reference past frames encoded at different resolutions, RFR provides seamless adaptive streaming, smoother rate control, and resilient real-time communications under fluctuating network conditions.

What is Reference Frame Resampling?

In legacy video codecs such as H.264 and standard profiles of HEVC, altering the video resolution requires inserting a new keyframe (an IDR frame). Keyframes do not reference prior frames and require significantly more data to encode, typically causing a noticeable bitrate spike and potential buffer underruns.

AV1 eliminates this constraint through Reference Frame Resampling. RFR enables inter-prediction between frames of differing spatial resolutions. The decoder rescales reference frames stored in the Decoded Picture Buffer (DPB) to match the dimensions of the current frame, allowing motion estimation and residual coding to function normally across resolution boundaries.

How Dynamic Resampling Operates

Reference Frame Resampling functions dynamically at the frame header level through a standardized pipeline:

  1. Resolution Signaling: Each frame header in AV1 carries its own width and height parameters. The encoder can decide dynamically—frame by frame—to change the encoding resolution based on network throughput, device load, or scene complexity.
  2. Buffer Scaling via Normative Filters: When a frame references an older frame with a different resolution, the decoder does not require external scaling. Instead, it utilizes AV1’s standardized 8-phase interpolation filters to rescale the reference frame internally to the target frame's dimensions.
  3. Motion Vector Scaling: Motion vectors are scaled proportionally to account for the spatial coordinate differences between the reference and the current frame. This ensures that spatial motion trajectories remain accurate even when moving between higher and lower pixel densities.
  4. Residual Processing: Once the reference frame is scaled and motion compensation is performed, the encoder subtracts the prediction from the source image, transforming and quantizing the residual data just as it would for a standard frame.

Practical Applications

Dynamic reference frame resampling is particularly beneficial in real-time scenarios such as WebRTC video conferencing, cloud gaming, and live broadcasting. When network bandwidth drops suddenly, an AV1 encoder can instantly drop the resolution of the next inter-frame to prevent packet loss and latency. Once bandwidth recovers, the encoder can upscale the resolution seamlessly, avoiding the disruptive bandwidth spikes associated with transmitting full keyframes.