How Netflix Uses VMAF to Measure AV1 Quality

Netflix evaluates the streaming efficiency and visual performance of the AV1 video codec by pairing extensive encoding tests with Video Multi-Method Assessment Fusion (VMAF), its open-source perceptual video quality metric. Instead of relying solely on traditional mathematical indicators like Peak Signal-to-Noise Ratio (PSNR), Netflix uses VMAF to simulate human perception and accurately quantify how much visual quality improves—or how much bandwidth is saved—when upgrading to AV1 from older standards like AVC (H.264), HEVC (H.265), and VP9.

The Role of VMAF in Perceptual Testing

Traditional objective metrics like PSNR and Structural Similarity Index (SSIM) calculate pixel-to-pixel differences, often penalizing advanced compression techniques that look visually identical to the human eye. AV1 introduces complex encoding tools such as directional intra-prediction, advanced transform block sizes, and film grain synthesis.

Because these tools alter pixels deliberately to optimize bandwidth, traditional metrics can report misleading quality drops. VMAF solves this by combining multiple quality metrics (such as Visual Information Fidelity and Detail Loss Metric) with a machine-learning model trained on subjective viewing scores. The result is a single metric, scaled from 0 to 100, that strongly correlates with human visual perception.

The Dynamic Optimizer and Convex Hull Analysis

To measure AV1's performance, Netflix integrates VMAF into its Dynamic Optimizer encoding framework. This process evaluates quality through the following steps:

1. Shot-Based Segmentation: Full-length content is split into individual video shots rather than fixed-length chunks. This accounts for distinct motion, grain, and lighting characteristics per scene.
2. Exhaustive Parameter Sweeps: Each shot is encoded across dozens of configurations, varying resolutions (from 360p up to 4K), quantization parameters, and AV1 encoder presets.
3. Convex Hull Generation: Netflix calculates the VMAF score for every encode. By plotting bitrate against VMAF, the system identifies the "convex hull"—the Pareto-optimal curve of operating points that yield the highest possible VMAF score at any targeted bitrate.

Quantifying Gains with Bjøntegaard Delta (BD) Metrics

To compare AV1 directly against existing codecs, Netflix utilizes the Bjøntegaard Delta Rate (BD-Rate) metric, calculated using VMAF as the quality axis rather than PSNR:

• BD-Rate (VMAF): Measures the average percentage of bandwidth saved when achieving the exact same VMAF score across comparison codecs.
• BD-VMAF: Measures the average gain in VMAF points when holding the streaming bitrate constant.

Using this methodology, Netflix demonstrated that AV1 delivers equivalent or superior VMAF scores with significant bandwidth reductions compared to previous generation codecs—consistently outperforming VP9 by 20% or more and AVC by up to 30–40%, particularly in bandwidth-constrained environments.

Evaluating Specific AV1 Features

VMAF enables Netflix to validate specific features built into the AV1 specification:

• Film Grain Synthesis: High film grain typically consumes massive bandwidth to encode cleanly. AV1 analyzes the grain, strips it out before compression, and transmits synthesis parameters so the decoder can recreate it locally. Netflix uses specialized VMAF variants to confirm that synthetically restored grain delivers the expected visual texture without scoring artifacts.
• Low-Bitrate Scaling: In mobile streaming conditions, AV1 avoids the heavy blocking and banding artifacts common in older codecs. VMAF curves verify that mobile streams maintain acceptable subjective quality thresholds (typically a VMAF score of 80 or above) at significantly lower bitrates.

By integrating VMAF directly into offline benchmarking pipelines, Netflix quantitatively confirms that the compute-intensive process of encoding in AV1 directly translates into higher visual fidelity and fewer playback interruptions for the end viewer.