Configure FFmpeg libx264 Adaptive Quantization

This article explains how to configure and fine-tune the adaptive quantization (AQ) settings in the libx264 encoder using FFmpeg. Adaptive quantization is a crucial encoding technique that redistributes bits within a frame to improve visual quality, particularly by reducing blocking artifacts in flat, dark, or low-complexity areas. You will learn the key parameters, their available modes, and how to apply them in your FFmpeg commands.

Understanding Adaptive Quantization Parameters

In libx264, adaptive quantization is controlled by two main parameters: AQ Mode (aq-mode) and AQ Strength (aq-strength).

To configure these parameters in FFmpeg, you should pass them via the -x264-params option, separating the arguments with a colon (:).

1. AQ Mode (aq-mode)

The AQ mode determines how the encoder identifies complex versus flat areas in a frame to redistribute bits.

• 0 (Disabled): Turns off adaptive quantization entirely.
• 1 (Variance AQ - Default): Distributes bits based on local variance. It allocates more bits to flat areas to prevent banding and blocking, while reducing bits in highly detailed areas where artifacts are harder for the human eye to see.
• 2 (Auto-Variance AQ): An experimental mode that attempts to calculate the optimal variance threshold per frame rather than using a fixed global average.
• 3 (Auto-Variance AQ with Bias to Dark Scenes): An improved version of mode 2 that biases bit distribution toward dark gradients and shadow details. This is highly recommended for cinematic content to prevent color banding in dark scenes.

2. AQ Strength (aq-strength)

The AQ strength controls the intensity of the bit redistribution.

• Default value: 1.0
• Range: Typically 0.0 to 1.5 (though values up to 2.0 are technically allowed).
• Lower values (< 1.0): Reduce the strength of the redistribution. Use this if you notice detailed, high-complexity textures (like grass or film grain) are becoming too blurry.
• Higher values (> 1.0): Increase the strength of the redistribution. Use this if you notice flat surfaces, skies, or dark scenes are suffering from heavy color banding or macroblocking.

FFmpeg Command Examples

To apply these settings, use the -x264-params flag in your FFmpeg command.

Example 1: Enabling Auto-Variance AQ with Default Strength This command sets AQ mode to 2 and keeps the default strength of 1.0:

ffmpeg -i input.mp4 -c:v libx264 -crf 23 -x264-params aq-mode=2:aq-strength=1.0 output.mp4

Example 2: Optimizing for Dark Scenes and Preventing Banding This command utilizes AQ mode 3 (dark-biased) with a slightly stronger redistribution rate of 1.3 to clean up blocky shadows:

ffmpeg -i input.mp4 -c:v libx264 -crf 20 -x264-params aq-mode=3:aq-strength=1.3 output.mp4

Example 3: Disabling Adaptive Quantization If you prefer a constant quantization parameter across the entire frame without any bit redistribution, disable AQ completely:

ffmpeg -i input.mp4 -c:v libx264 -crf 23 -x264-params aq-mode=0 output.mp4