How to Use overlay_cuda in FFmpeg on NVIDIA GPUs

Hardware-accelerated picture-in-picture (PiP) allows you to composite multiple video streams entirely within your NVIDIA GPU's VRAM, eliminating CPU bottlenecks and system memory transfer overhead. This guide explains how to set up and execute an end-to-end GPU-accelerated video overlay workflow in FFmpeg using the overlay_cuda filter alongside hardware decoding and encoding.

Prerequisites

To use overlay_cuda, you need an NVIDIA graphics card with current proprietary drivers and an FFmpeg build compiled with the following flags enabled:

  • --enable-cuda-nvcc (or --enable-cuda-llvm)
  • --enable-cuvid
  • --enable-nvenc
  • --enable-nonfree
  • --enable-libnpp

The Complete GPU Pipeline

For maximum performance, the entire pipeline—decoding, scaling, compositing, and encoding—should remain on the GPU. If you upload frames back and forth to system RAM, performance drops significantly.

Run the following command to place a smaller secondary video in the bottom-right corner of a primary video:

ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i main_video.mp4 \
       -hwaccel cuda -hwaccel_output_format cuda -i pip_video.mp4 \
       -filter_complex "[1:v]scale_cuda=640:360[pip];[0:v][pip]overlay_cuda=x=W-w-30:y=H-h-30[out]" \
       -map "[out]" -map 0:a? \
       -c:v h264_nvenc -preset p4 -b:v 5M \
       -c:a copy output.mp4

Command Breakdown

  • -hwaccel cuda -hwaccel_output_format cuda: Tells FFmpeg to decode the input files using NVDEC and keep the decoded frames directly in CUDA device memory. This must precede each -i input flag.
  • scale_cuda=640:360: Resizes the secondary input on the GPU. Using the software scale filter here would cause an error because the frames reside in CUDA memory, requiring a hardware-compatible scaling filter like scale_cuda or scale_npp.
  • overlay_cuda=x=W-w-30:y=H-h-30: Blends the scaled overlay stream ([pip]) onto the background stream ([0:v]).
    • W and H represent the width and height of the main video.
    • w and h represent the width and height of the overlay video.
    • 30 defines the pixel padding from the right and bottom edges.
  • -c:v h264_nvenc: Encodes the final output video using the NVENC hardware encoder directly from CUDA memory.
  • -map 0:a? -c:a copy: Copies the primary video's audio track without re-encoding, avoiding errors if no audio stream exists.

Mixing Software Decoded Inputs with overlay_cuda

If one of your input files uses an unsupported codec that NVDEC cannot decode, you can manually upload software frames to CUDA memory using hwupload_cuda:

ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i main_video.mp4 \
       -i software_pip.mkv \
       -filter_complex "[1:v]hwupload_cuda,scale_cuda=480:270[pip];[0:v][pip]overlay_cuda=x=20:y=20[out]" \
       -map "[out]" -c:v hevc_nvenc output.mp4

In this setup, the secondary input is decoded via the CPU, uploaded to VRAM with hwupload_cuda, scaled, and then composited via overlay_cuda at the top-left coordinate (20, 20).