Difference-Modulated Masked Prediction in AV1
This article provides a technical overview of difference-modulated masked prediction within the AV1 video codec's inter coding framework. It examines how this advanced tool functions as part of AV1's compound prediction toolset, detailing the underlying mechanics of how prediction masks are derived from pixel differences, the coding efficiency benefits it introduces, and how it handles complex motion scenarios such as occlusions and lighting transitions.
Compound Prediction in AV1
Inter coding in video compression reduces temporal redundancy by predicting blocks of pixels from previously decoded reference frames. AV1 extends traditional single-reference prediction by offering compound prediction, where two distinct motion vectors point to two different reference frames to produce two prediction signals, \(P_1\) and \(P_2\).
While standard compound prediction simply computes a uniform average (50/50 blend) or a frame-distance-weighted average of the two signals, this approach often fails near motion boundaries, occlusions, or areas with dynamic lighting changes. To address this, AV1 incorporates masked compound prediction, which applies sample-by-sample weights using a 2D spatial mask.
What is Difference-Modulated Masked Prediction?
Difference-modulated masked prediction (often referred to as
difference-weighted prediction or COMPOUND_DIFFWTD) is a
specific type of masked compound prediction. Unlike wedge-based
prediction, which selects a predefined geometric partition from a fixed
codebook, difference-modulated prediction calculates the blending mask
dynamically from the prediction signals themselves.
The core principle is that the absolute difference between the two predictor signals provides information about which predictor is more reliable or where a boundary lies.
Mask Derivation Process
The generation of the blending mask operates as follows:
Pixel-Wise Difference Calculation: For every sample \((x, y)\) in the block, the encoder and decoder compute the absolute difference between the two prediction blocks: \[\Delta(x, y) = |P_1(x, y) - P_2(x, y)|\]
Non-Linear Mapping: The calculated difference \(\Delta(x, y)\) is mapped to a weight value using a predefined lookup table or thresholding function. When the difference between the two predictors is small, the mask assigns nearly equal weights to both signals, smoothing out random noise. When the difference is substantial, indicating a boundary, edge, or occlusion, the mask shifts the weight heavily toward one designated predictor.
Sample Blending: The final predicted pixel value \(P(x, y)\) is computed using the derived mask \(M(x, y)\): \[P(x, y) = \frac{M(x, y) \cdot P_1(x, y) + (64 - M(x, y)) \cdot P_2(x, y)}{64}\] (Note: AV1 typically uses a 6-bit representation where weights sum to 64).
Signaling and Decoder Operation
A primary advantage of difference-modulated masked prediction is its low signaling overhead. Because the mask is derived deterministically from the reconstructed motion-compensated samples \(P_1\) and \(P_2\), the decoder can construct the exact same mask independently.
The bitstream only needs to signal:
- The decision to use difference-modulated compound mode instead of uniform or wedge prediction.
- Which of the two predictors serves as the primary base predictor (determining whether large differences favor \(P_1\) or \(P_2\)).
No pixel-by-pixel mask data or shape indices need to be transmitted, preserving bitstream bandwidth.
Key Advantages in Compression
- Edge and Occlusion Handling: When an object moves across a background, standard averaging creates ghosting artifacts along the edges. Difference-modulated masking naturally detects these discrepancies and suppresses the inaccurate reference signal.
- Gradual Transitions: It adapts smoothly to localized lighting shifts, shadows, and non-rigid motion where uniform geometric partitions (like wedge modes) are too rigid to fit the boundary accurately.
- Residual Reduction: By yielding a significantly closer match to the original block, it minimizes high-frequency residual energy, allowing the transform and quantization stages to encode residual blocks with far fewer bits.