Configure Psycho-Visual Quantization in FFmpeg libx264

This article explains how to configure and fine-tune psycho-visual quantization strength in the libx264 encoder using FFmpeg. You will learn the specific command-line parameters to adjust these settings, understand how the underlying psycho-visual rate-distortion (psy-RD) and psycho-visual Trellis (psy-trellis) systems work, and find recommended configurations for different types of video content.

Understanding Psycho-Visual Options in libx264

Psycho-visual optimization biases the encoder toward visual quality rather than strict mathematical metrics like PSNR (Peak Signal-to-Noise Ratio). It helps retain fine details, film grain, and textures that human eyes notice, preventing the encoder from smoothing them out to save bitrate.

In libx264, this is controlled via two parameters passed as a pair: 1. Psy-RD (Psycho-visual Rate-Distortion): Controls the strength of detail preservation. 2. Psy-Trellis (Psycho-visual Trellis): Controls the strength of detail preservation during the final quantization steps.

The ffmpeg Command Syntax

To configure these strengths, you must pass the psy-rd parameter using the -x264-params or -x264opts flag. The format is psy-rd=psy-rd_strength:psy-trellis_strength.

Here is a standard FFmpeg command template:

ffmpeg -i input.mp4 -c:v libx264 -crf 20 -x264-params psy-rd=1.0:0.15 output.mp4

In this command, 1.0 is the strength of the Psy-RD, and 0.15 is the strength of the Psy-Trellis.

Parameter Breakdown

• Psy-RD (First Value, range: 0.0 to 10.0+):
  • The default value is 1.0 for most presets.
  • Increasing this value forces the encoder to keep complex textures and grain, though setting it too high can introduce blockiness in flat areas if the bitrate is too low.
  • Setting it to 0.0 completely disables psycho-visual optimizations, which is useful only for PSNR/SSIM benchmarking.
• Psy-Trellis (Second Value, range: 0.0 to 10.0+):
  • The default value is 0.0 for most presets (disabled).
  • Enabling Psy-Trellis (typically between 0.10 and 0.25) helps preserve very fine textures and grain.
  • Setting this value too high can cause unwanted ringing artifacts around sharp edges and may significantly decrease encoding speed.

Recommended Configurations

You do not always need to manually set these values; the -tune flags in libx264 configure them automatically. However, manual tweaking is beneficial for custom encodes:

• Film and Live-Action (High Detail/Grain):

  -x264-params psy-rd=1.0:0.15

  This is the default setting used by -tune film. It balances texture preservation with encoding efficiency.

• Heavy Film Grain (High-Bitrate Archives):

  -x264-params psy-rd=1.0:0.25

  Slightly higher Psy-Trellis to lock in fine grain patterns, best paired with a lower CRF (higher quality) like 16 to 18.

• Animation and Flat Vectors:

  -x264-params psy-rd=0.4:0.0

  Animation benefits from less psycho-visual optimization because flat surfaces can easily become blocky or noisy when Psy-RD is too high. This is the setting applied when using -tune animation.

• Disabling Psycho-Visual Optimizations (For Metrics Only):

  -x264-params psy=0

  Alternatively, setting psy-rd=0.0:0.0 will turn off the system entirely, lowering visual quality but maximizing mathematical encoding metrics.