Compiling FFmpeg with NVIDIA NVENC on Linux

Compiling FFmpeg with NVIDIA hardware acceleration enables high-performance, GPU-accelerated video encoding and decoding on Linux systems. By leveraging the NVIDIA Video Codec SDK (NVENC/NVDEC) alongside CUDA, you can dramatically reduce CPU usage and speed up media processing workflows. This guide covers the entire end-to-end process, from installing the required NVIDIA driver and SDK dependencies to configuring and building the custom FFmpeg binary.

Prerequisites and Driver Installation

Before compiling FFmpeg, your system must have the proprietary NVIDIA drivers and the CUDA Toolkit installed.

Install NVIDIA Drivers and CUDA: Use your distribution’s package manager to install the latest stable proprietary drivers and CUDA toolkit. For example, on Ubuntu or Debian systems:

sudo apt update
sudo apt install nvidia-driver-535 cuda-toolkit-12-1

Verify the Installation: Ensure your GPU is properly recognized and the drivers are active by running:

nvidia-smi

Installing Build Essential Tools

Building FFmpeg from source requires standard compilation tools and core libraries. Install these base packages to prevent build errors:

sudo apt install build-essential yasm nasm cmake git pkg-config libuname-arm64-dev

Cloning and Installing the NVIDIA Headers

FFmpeg interfaces with NVIDIA hardware via the hardware abstraction headers. These headers must be installed manually from the official FFmpeg repository.

Clone the ffnvcodec repository:

git clone https://git.videolan.org/git/ffmpeg/nv-codec-headers.git

Compile and install the headers:

cd nv-codec-headers
make
sudo make install
cd ..

Downloading the FFmpeg Source Code

Clone the official FFmpeg source code repository to get the latest stable features and security patches.

git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg
cd ffmpeg

Configuring the Compilation Flags

To enable NVIDIA hardware acceleration, you must explicitly pass the CUDA and NVENC flags to the ./configure script. You also need to ensure that pkg-config can locate the ffnvcodec headers installed in the previous step.

Run the configuration script with the following parameters:

./configure \
  --enable-nonfree \
  --enable-gpl \
  --enable-cuda-nvcc \
  --enable-libnpp \
  --extra-cflags=-I/usr/local/cuda/include \
  --extra-ldflags=-L/usr/local/cuda/lib64 \
  --disable-static \
  --enable-shared

--enable-nonfree and --enable-gpl: Required because NVIDIA’s SDK components are proprietary.
--enable-cuda-nvcc: Enables the CUDA compiler for hardware-accelerated filters.
--enable-libnpp: Enables NVIDIA Performance Primitives for hardware video scaling and pixel format conversion.

Compiling and Installing FFmpeg

Once the configuration finishes without errors, compile the source code using multiple CPU cores to speed up the process, then install the binaries.

make -j$(nproc)
sudo make install
sudo ldconfig

The ldconfig command updates the system links to ensure the newly compiled shared libraries are recognized globally.

Verifying Hardware Acceleration Support

To confirm that your custom-built FFmpeg successfully supports NVIDIA hardware acceleration, query the available encoders and decoders.

Check for NVENC (Encoding):

ffmpeg -encoders | grep nvenc

This should return lines for encoders like h264_nvenc and hevc_nvenc. * Check for NVDEC (Decoding):

ffmpeg -decoders | grep cuvid

You can now use these hardware-accelerated codecs in your FFmpeg commands by specifying flags like -c:v h264_nvenc to offload video encoding tasks to your NVIDIA GPU.