Using NVIDIA NVENC with FFmpeg on Linux

This guide provides a comprehensive, step-by-step walkthrough for leveraging NVIDIA NVENC hardware acceleration to encode videos using FFmpeg on a Linux system. You will learn how to verify your hardware compatibility, install the necessary drivers and dependencies, and execute optimized FFmpeg commands for high-performance H.264 and HEVC (H.265) video encoding.

Prerequisites and System Verification

Before configuring FFmpeg, you must ensure that your system has a compatible NVIDIA graphics card and the correct proprietary drivers installed. NVENC is supported on most Kepler-generation (GTX 600 series) and newer GPUs.

To verify that your NVIDIA GPU is recognized and check the installed driver version, open your terminal and run:

nvidia-smi

This command should output a table displaying your GPU model and driver version. If it returns an error, you must install the proprietary NVIDIA drivers via your distribution’s package manager before proceeding.

Installing FFmpeg with NVENC Support

Most modern Linux distributions include NVENC support in their default FFmpeg packages, provided the NVIDIA headers are available during compilation.

Ubuntu and Debian-based Systems

On Ubuntu 20.04 and newer, the standard FFmpeg package comes compiled with NVENC support out of the box. Install it alongside the required libraries using:

sudo apt update
sudo apt install ffmpeg libavcodec-extra

Arch Linux

Arch Linux users can install FFmpeg directly from the official repositories, which includes NVENC support by default:

sudo pacman -S ffmpeg

Verifying NVENC Availability

To confirm that your FFmpeg installation can access the NVIDIA hardware encoders, list the available H.264 and HEVC encoders with the following commands:

ffmpeg -encoders | grep nvenc

If configured correctly, you will see h264_nvenc and hevc_nvenc listed in the output.

Essential FFmpeg NVENC Commands

Once verified, you can begin offloading your video encoding tasks from your CPU to your NVIDIA GPU.

Basic H.264 Encoding

To encode a video to H.264 using NVENC with default settings, use the h264_nvenc codec:

ffmpeg -i input.mp4 -c:v h264_nvenc -c:a copy output.mp4

Basic HEVC (H.265) Encoding

For better compression efficiency, you can use the HEVC encoder:

ffmpeg -i input.mp4 -c:v hevc_nvenc -c:a copy output.mp4

Advanced Customization and Quality Control

To achieve the best balance between file size and visual quality, you should customize the bitrate, quality targets, and performance presets.

Constant Quality Mode (CQ)

Instead of a fixed bitrate, use Constant Quality mode to let the encoder dynamically adjust bitrate based on scene complexity. This is controlled via the -cq flag (lower values mean higher quality, typically between 19 and 23 is ideal):

ffmpeg -i input.mp4 -c:v h264_nvenc -rc constqp -qp 23 output.mp4

Alternatively, you can use the Variable Bitrate (VBR) mode with a target quality lookahead:

ffmpeg -i input.mp4 -c:v h264_nvenc -rc vbr -cq 23 -b:v 0 output.mp4

Performance Presets

NVENC offers several presets ranging from fastest performance to maximum quality. These are defined using the -preset flag, with options including p1 (fastest) through p7 (slowest/highest quality):

ffmpeg -i input.mp4 -c:v h264_nvenc -preset p6 -rc vbr -cq 22 -b:v 0 output.mp4

Hardware-Accelerated Decoding and Scaling

To achieve maximum throughput, you can also offload the video decoding and resizing to the GPU using NVIDIA’s NVDEC and CUVID filters:

ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 -vf "scale_cuda=1920:1080" -c:v h264_nvenc -preset p5 output.mp4

This pipeline keeps the video data entirely within the GPU memory from decoding to encoding, significantly reducing CPU utilization and maximizing processing speed.