How to Compile FFmpeg with NVENC on Windows

Compiling FFmpeg with NVIDIA NVENC support on Windows allows you to leverage your NVIDIA graphics card for hardware-accelerated video encoding, significantly reducing rendering times. This guide provides a direct, step-by-step walkthrough to set up a MSYS2 compilation environment, install the necessary NVIDIA header files, and build a custom FFmpeg binary with NVENC enabled.

Step 1: Install MSYS2 and Build Tools

MSYS2 provides the build environment and compiler toolchain required to compile FFmpeg on Windows.

Download and run the installer from the official MSYS2 website.
Once installed, open the MSYS2 MinGW 64-bit terminal from your Start Menu.
Update the package database and core system packages by running:
```
pacman -Syu
```
(If the terminal closes, reopen MSYS2 MinGW 64-bit and run the command again to complete updates).
Install the required build tools, compilers, and git by running:
```
pacman -S --needed base-devel mingw-w64-x86_64-toolchain git yasm nasm pkg-config
```
Confirm the installation by pressing Y when prompted.

Step 2: Install NVIDIA Web Codec Headers (ffnvcodec)

FFmpeg requires the interface headers for the NVIDIA Command Line Interface (CLI) to enable NVENC. These are maintained by the FFmpeg community in the ffnvcodec repository.

Clone the repository:

git clone https://git.videolan.org/git/ffmpeg/nv-codec-headers.git

Navigate into the cloned directory:
```
cd nv-codec-headers
```
Compile and install the headers to the MinGW 64-bit path:
```
make install PREFIX=/mingw64
```
Move back to your home directory:
```
cd ..
```

Step 3: Clone the FFmpeg Source Code

Clone the official FFmpeg source code repository to your local machine:

git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg
cd ffmpeg

Step 4: Configure the Build

To compile FFmpeg with NVENC, you must configure the build flag to enable non-free code and the specific NVIDIA hardware acceleration libraries.

Run the configuration script:

./configure --enable-nonfree --enable-nvenc --enable-nvdec --enable-ffnvcodec --enable-gpl

Note: If you want a static build (a single, portable .exe file), you can add --disable-shared --enable-static to the configure command.

Step 5: Compile and Install FFmpeg

Start the compilation process. Use the -j flag followed by the number of CPU cores you want to allocate to speed up compile time:
```
make -j$(nproc)
```
Once the compilation completes, verify that the binary was created successfully by checking the version:
```
./ffmpeg.exe -version
```

Step 6: Verify NVENC Support

To confirm that NVENC is successfully compiled into your FFmpeg binary, list the available hardware-accelerated encoders:

./ffmpeg.exe -encoders | grep nvenc

You should see output listing h264_nvenc and hevc_nvenc (h265), confirming that FFmpeg is ready to use your NVIDIA GPU for hardware-accelerated video encoding. Your compiled ffmpeg.exe file can be found inside the ffmpeg directory.