How to Configure h264_vaapi Rate Control in FFmpeg

This article explains how to configure rate control and quality options using the hardware-accelerated h264_vaapi encoder in FFmpeg. You will learn how to set up different rate control modes—including Constant QP (CQP), Constant Bitrate (CBR), Variable Bitrate (VBR), and Intelligent Constant Quality (ICQ)—and adjust compression levels to balance encoding speed and output quality on Intel and AMD graphics hardware.

Setting Up the VA-API Hardware Device

Before applying rate control options, you must initialize the VA-API hardware device and upload your input video to the GPU frame memory. Use the following baseline command structure:

ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input.mp4 -c:v h264_vaapi [encoding_options] output.mp4

Configuring Rate Control Modes

The h264_vaapi encoder determines how to allocate bitrate using the -rc_mode flag or specific bitrate and quality parameters.

1. Constant QP (CQP)

Constant QP mode encodes every frame with a fixed quantization parameter. This mode prioritizes image quality over file size.

How to configure: Set -rc_mode to CQP (or 1) and define the quality level using -global_quality.
Scale: Values range from 1 (highest quality, largest file size) to 51 (lowest quality, smallest file size). A value between 20 and 26 is recommended for general use.

ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input.mp4 \
  -c:v h264_vaapi -rc_mode CQP -global_quality 23 output.mp4

2. Variable Bitrate (VBR)

VBR dynamically adjusts the bitrate based on the complexity of the video scene, preserving bandwidth during static scenes and increasing it during high-motion scenes.

How to configure: Set -rc_mode to VBR (or 2), define the target average bitrate with -b:v, and specify a maximum peak bitrate with -maxrate.

ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input.mp4 \
  -c:v h264_vaapi -rc_mode VBR -b:v 5M -maxrate 10M output.mp4

3. Constant Bitrate (CBR)

CBR maintains a strict, consistent output bitrate regardless of scene complexity. This mode is ideal for live streaming and network-constrained environments.

How to configure: Set -rc_mode to CBR (or 3), and match the target bitrate (-b:v), minimum bitrate (-minrate), and maximum bitrate (-maxrate) to the same value. You should also define the buffer size (-bufsize).

ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input.mp4 \
  -c:v h264_vaapi -rc_mode CBR -b:v 4M -minrate 4M -maxrate 4M -bufsize 8M output.mp4

4. Intelligent Constant Quality (ICQ)

Supported primarily on Intel GPUs, ICQ is a content-adaptive rate control mode similar to Constant Rate Factor (CRF). It attempts to maintain a consistent subjective quality level while optimizing compression efficiency.

How to configure: Set -rc_mode to ICQ and define the target quality level with -global_quality.

ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input.mp4 \
  -c:v h264_vaapi -rc_mode ICQ -global_quality 23 output.mp4

Configuring Quality and Speed Trade-offs

You can adjust the speed of the hardware encoder and the resulting image quality by tuning the compression level, profiles, and GOP structure.

Compression Level

Use the -compression_level flag to trade encoding speed for compression efficiency: * Scale: Typically ranges from 1 to 7. * Setting 1: Provides the highest quality and best compression, but uses the most GPU processing time. * Setting 7: Provides the fastest encoding speed, but results in lower compression efficiency (larger files or lower quality at a fixed bitrate).

ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input.mp4 \
  -c:v h264_vaapi -rc_mode VBR -b:v 5M -compression_level 1 output.mp4

Profiles and Levels

Explicitly setting the H.264 profile and level ensures compatibility with target playback devices: * Use -profile:v to set the profile (e.g., high, main, or constrained_baseline). * Use -level to set the H.264 level (e.g., 41 for Level 4.1).

ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input.mp4 \
  -c:v h264_vaapi -profile:v high -level 41 output.mp4