How AV1 Overlay Frames Work in Two-Pass Encoding

In an AV1 two-pass compression pipeline, overlay frames work in tandem with Alternate Reference (AltRef) frames to improve visual quality and compression efficiency without confusing the display pipeline. During the first pass, the encoder analyzes the temporal sequence to schedule forward-looking reference frames, while the second pass encodes a filtered, invisible lookahead frame followed later by an overlay frame at the true display timestamp. This mechanism allows AV1 to leverage bidirectional temporal filtering and complex inter-frame prediction while cleanly outputting the correct display sequence for decoders.

The Role of Alternate Reference (AltRef) Frames

To understand overlay frames, one must first understand AltRef frames. In AV1, an AltRef frame is an artificially constructed frame created by applying temporal filtering across multiple neighboring frames to dramatically reduce temporal noise.

Because an AltRef frame is heavily filtered, it represents a synthetic future point in time. The encoder marks this frame with a show_frame = 0 flag. The decoder stores this frame in one of its eight reference frame buffers, but it does not output it to the display. Subsequent frames can then use this high-quality, denoised reference for forward and backward prediction.

First Pass: Lookahead and Frame Planning

In the first pass of a two-pass encoding pipeline, the encoder rapidly processes the video to gather statistical data about motion vectors, scene complexity, texture, and scene changes.

During this phase, the rate control algorithm:

• Divides the sequence into mini-GOPs (Groups of Pictures).
• Identifies optimal temporal locations for AltRef frames.
• Determines the exact presentation timestamps where those AltRef frames must eventually be presented to the viewer.
• Budgets the bit distribution, allocating higher bitrates to reference frames and minimal bitrates to overlay frames.

Second Pass: Execution and Overlay Delivery

During the second pass, the encoder applies the decisions made in the first pass using full rate-distortion optimization (RDO).

1. AltRef Encoding: The encoder constructs the temporally filtered AltRef frame and transmits it early in the bitstream as an invisible frame (show_frame = 0).
2. Intermediate Frame Encoding: The encoder codes intermediate B-frames and P-frames, referencing both past frames and the future AltRef frame in the buffer.
3. The Overlay Frame: When the timeline reaches the actual display time of the AltRef frame, the encoder emits the overlay frame.

The overlay frame can function in two distinct ways depending on the encoder's RDO decision:

• Show-Existing-Frame: If the temporal filtering applied to the AltRef did not create noticeable distortion or ghosting relative to the original source, the overlay frame simply acts as a display command (show_existing_frame = 1). This command instructs the decoder to output the AltRef frame directly from its reference buffer. This operation requires virtually zero bits.
• Residual Inter-Frame (Standard Overlay): If the AltRef frame exhibits filtering artifacts or motion blur compared to the real source frame at that timestamp, the encoder codes an overlay frame as an actual inter-coded frame. It uses the AltRef frame as its primary predictor and encodes only the small residual error (differences) required to restore sharpness and detail. Once reconstructed, this frame is displayed to the viewer (show_frame = 1) and can optionally refresh the reference buffer.

Efficiency Gains

By separating the reference utility from the display mechanism, the two-pass pipeline avoids allocating heavy bit budgets to display-only frames. The heavy visual processing is offloaded to the AltRef frame, which serves multiple surrounding frames as a reference, while the overlay frame ensures accurate visual reconstruction at near-zero bitrate overhead.