Configure h264_nvenc Rate Control and Quality in FFmpeg

This article provides a practical guide on configuring the rate control and quality options for NVIDIA’s hardware-accelerated H.264 encoder (h264_nvenc) in FFmpeg. You will learn how to select different rate control modes, set target bitrates, configure quality-based variable bitrates, use presets, and apply advanced tuning parameters to optimize your video encoding workflow.

Understanding Rate Control Modes (-rc)

The rate control mode determines how the encoder allocates bits to maintain either a specific bitrate or a specific quality level. You can set the rate control mode using the -rc option.

1. Constant Quality (VBR with Target Quality)

For most archival and general-purpose encoding, a quality-based variable bitrate (VBR) is recommended. This behaves similarly to the CRF (Constant Rate Factor) mode in software encoders.

To enable this, set -rc to vbr, set the target bitrate -b:v to 0, and specify a quality level using -cq (Constant Quality).

ffmpeg -i input.mp4 -c:v h264_nvenc -rc vbr -cq 23 -b:v 0 output.mp4

• -cq 23: Controls the visual quality. Range is 1 to 51, where lower values mean better quality and larger file sizes. A value between 19 and 23 is a good balance for H.264.

2. Constant Bitrate (CBR)

CBR is ideal for live streaming where bandwidth is strictly limited. It forces the encoder to maintain a consistent output bitrate.

ffmpeg -i input.mp4 -c:v h264_nvenc -rc cbr -b:v 5M -maxrate 5M -bufsize 10M output.mp4

• -b:v 5M: Sets the target bitrate to 5 Mbps.
• -maxrate 5M: Restricts the maximum bitrate to match the target.
• -bufsize 10M: Sets the receiver buffer size, which controls rate fluctuations.

3. Variable Bitrate (VBR with Bitrate Cap)

This mode targets an average bitrate but allows the encoder to use more bits for complex scenes and fewer bits for simple scenes.

ffmpeg -i input.mp4 -c:v h264_nvenc -rc vbr -b:v 4M -maxrate 8M -bufsize 16M output.mp4

• -b:v 4M: The average target bitrate.
• -maxrate 8M: The maximum allowed bitrate peak.

4. Constant QP (Constant Quantization Parameter)

Constant QP bypasses any bitrate-targeting logic and encodes every frame at a fixed quantization level. It is computationally fast but highly inefficient for file size.

ffmpeg -i input.mp4 -c:v h264_nvenc -rc constqp -qp 20 output.mp4

• -qp 20: Applies QP 20 globally. You can also specify different QP values for I, P, and B frames using -init_qp_i, -init_qp_p, and -init_qp_b.

Encoder Presets and Tuning

NVIDIA’s modern NVENC SDK (version 11 and newer) utilizes a preset naming convention from p1 to p7 to manage the trade-off between encoding speed and quality.

Presets (-preset)

• p1 to p2: Fastest encoding speed, lowest quality (ideal for low-latency streaming on weak hardware).
• p4: Medium speed and quality (default).
• p6 to p7: Slowest encoding speed, highest quality (ideal for high-quality archiving).

ffmpeg -i input.mp4 -c:v h264_nvenc -preset p6 -rc vbr -cq 22 -b:v 0 output.mp4

Tuning (-tune)

The -tune parameter configures internal encoder settings for specific use cases: * hq: High Quality (default). * ll: Low Latency (optimizes for real-time applications). * ull: Ultra Low Latency (removes B-frames to reduce delay).

Advanced Quality Enhancements

To maximize the visual quality of your NVENC encodes, you can enable additional hardware-assisted features.

Look-Ahead (-rc-lookahead)

Enables the encoder to analyze future frames (up to 32) to optimize bitrate distribution and B-frame placement. This significantly improves quality during fast-motion scenes.

ffmpeg -i input.mp4 -c:v h264_nvenc -rc vbr -cq 22 -b:v 0 -rc-lookahead 32 output.mp4

Adaptive Quantization (-spatial-aq and -temporal-aq)

• Spatial AQ (-spatial-aq 1): Adjusts QP values within a single frame. It allocates more bits to complex, textured areas where compression artifacts are easily noticed.
• Temporal AQ (-temporal-aq 1): Adjusts QP values across multiple frames. It reduces blockiness in fast-moving scenes by tracking motion.

ffmpeg -i input.mp4 -c:v h264_nvenc -rc vbr -cq 22 -b:v 0 -spatial-aq 1 -temporal-aq 1 output.mp4

Multipass Encoding (-multipass)

Multipass encoding improves rate control precision across the video. Modern NVENC supports high-speed hardware multipass modes: * disabled (or 0): Single-pass encoding. * qres: Two-pass encoding, where the first pass runs at quarter-resolution (recommended for speed and quality balance). * fullres: Two-pass encoding, where the first pass runs at full resolution (highest quality).

ffmpeg -i input.mp4 -c:v h264_nvenc -rc vbr -b:v 5M -multipass fullres output.mp4