Psychovisual Rate Distortion in AV1 Encoders

This article explains the concept of psychovisual rate distortion optimization (Psy-RDO) and its practical implementation within modern AV1 video encoders such as SVT-AV1 and libaom. While traditional rate-distortion optimization relies on purely mathematical error metrics that often introduce visual softness, psychovisual rate distortion models the human visual system to preserve textures, edges, and perceived clarity. The following sections explore how classical optimization falls short, the mechanics behind perceptual distortion modeling, and the specific algorithms AV1 encoders use to balance visual quality against bitrate.

Understanding Classical Rate-Distortion Optimization (RDO)

In digital video compression, an encoder must constantly choose between multiple coding options—such as block partition sizes, prediction modes, and transform types. Rate-Distortion Optimization (RDO) is the decision-making framework used to select the best option using the Lagrangian cost function:

\[J = D + \lambda R\]

In this equation, \(R\) represents the rate (the number of bits required), \(D\) represents the distortion (the difference between the original and reconstructed block), and \(\lambda\) (lambda) is the Lagrange multiplier balancing the two based on the target quantizer.

Historically, encoders measure distortion (\(D\)) using mathematical metrics like Mean Squared Error (MSE) or Sum of Absolute Differences (SAD). While computationally efficient, these metrics assume that all pixel-level errors are equally noticeable to the human eye. In reality, MSE heavily penalizes high-frequency deviations, leading the encoder to favor smoothed, blurred reconstructions that minimize raw numerical variance over textured reconstructions that appear sharper to human observers.

What is Psychovisual Rate Distortion?

Psychovisual rate distortion incorporates principles of the Human Visual System (HVS) directly into the RDO cost calculation. Instead of treating every pixel deviation equally, Psy-RDO adjusts the distortion term \(D\) (or biases the cost \(J\)) based on perceptual impact:

• Spatial Masking: Artifacts and noise are far less perceptible in busy, high-texture areas (like foliage or water) than in smooth gradients (like clear skies or skin tones).
• Contrast Sensitivity: The human eye is more sensitive to structural degradations and edge alignment than to subtle global luminance shifts.
• Temporal Masking: Rapidly moving objects or scene cuts reduce the eye's resolving power, allowing more compression artifacts to pass unnoticed compared to static scenes.

By modeling these factors, Psy-RDO allows higher mathematical error in areas where the eye cannot perceive it, saving bits to preserve fine edges, grain, and high-frequency structural details elsewhere.

Implementation of Psy-RDO in AV1 Encoders

AV1 encoders, predominantly SVT-AV1 and libaom, implement psychovisual optimization across multiple stages of the encoding pipeline.

1. Perceptual Block Variance and Adaptive Quantization

Before mode evaluation occurs, AV1 encoders analyze spatial complexity using variance-based adaptive quantization (often termed aq-mode in AV1 implementations):

• Flatness Detection: Low-variance blocks are assigned lower quantization parameters (finer quantization) to prevent banding and blockiness in smooth areas.
• Texture Weighting: Highly textured blocks receive slightly higher quantization parameters because the natural chaos of the texture masks slight quantization noise.

2. Psy-RDO in Transform and Mode Decision

During the exhaustive evaluation of prediction modes and transform sizes, encoders like SVT-AV1 employ explicit Psy-RDO scoring heuristics.

Instead of evaluating \(D\) purely as MSE, the distortion metric is blended with an energy-matching or cross-correlation term:

\[D_{psy} = D_{MSE} - \beta \cdot C_{energy}\]

Here, \(C_{energy}\) measures how well the reconstructed block preserves the AC energy (high-frequency detail) of the original source block, and \(\beta\) is a tuning coefficient. If an encoding mode completely smooths out fine grain or sharp lines, \(C_{energy}\) drops, increasing \(D_{psy}\) and causing the encoder to discard that mode in favor of one that retains texture, even if the raw bit cost is higher.

3. Transform Domain Psychovisual Weighting

AV1 supports multiple transform types, including DCT, asymmetric discrete sine transforms (ADST), and identity transforms across sizes from 4x4 to 64x64. Encoders adjust the distortion calculation in the frequency domain:

• Coefficients corresponding to high spatial frequencies are scaled based on Contrast Sensitivity Function (CSF) curves.
• Quantization matrices (QM) in AV1 scale quantization steps per frequency band, ensuring that visually critical mid-to-low frequencies are prioritized over frequencies beyond normal human visual acuity.

4. Temporal Psy-RDO via Lookahead

AV1 encoders leverage deep lookahead buffers to evaluate temporal dependencies. In SVT-AV1, temporal RDO tracks how well details persist across frames:

• Static or slow-panning backgrounds are given high visual priority because the human eye has time to track and inspect stationary details.
• In chaotic scenes with high motion, temporal Psy-RDO raises the quantization parameter, accepting lower fidelity per frame because the temporal masking effect of the human eye hides the degraded detail.

Impact on Metrics and Subjective Quality

Enabling psychovisual rate distortion in AV1 encoders alters traditional quality benchmarks. Because Psy-RDO deliberately accepts higher mathematical deviation to preserve visual structure, traditional metrics like PSNR (Peak Signal-to-Noise Ratio) almost invariably decrease.

However, subjective viewing tests and perception-oriented metrics—such as SSIM, VMAF (Video Multi-Method Assessment Fusion), and butteraugli—consistently show substantial improvements. The result is a stream that eliminates plastic-looking skin, retains natural film grain, and prevents muddy motion artifacts at identical or lower bitrates.