FFmpeg overlay_cuda Guide for NVIDIA GPU PiP

This article explains how to use the overlay_cuda filter in FFmpeg to create hardware-accelerated picture-in-picture (PiP) video overlays using NVIDIA GPUs. You will learn the necessary prerequisites, the core command-line syntax, and how to keep the entire decoding, scaling, compositing, and encoding pipeline on the GPU to maximize performance and minimize CPU overhead.

Prerequisites

To use overlay_cuda, your system must meet the following requirements: * An NVIDIA GPU that supports NVDEC (hardware decoding) and NVENC (hardware encoding). * NVIDIA proprietary graphics drivers installed. * A build of FFmpeg compiled with CUDA support (--enable-cuda-nvcc and --enable-libnpp).

The FFmpeg Command

To perform hardware-accelerated picture-in-picture processing, you must decode both input videos into CUDA memory, scale the overlay video (if necessary) on the GPU, apply the overlay_cuda filter, and then encode the output using h264_nvenc or hevc_nvenc.

Here is the standard command template:

ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i background.mp4 \
       -hwaccel cuda -hwaccel_output_format cuda -i pip_video.mp4 \
       -filter_complex "[1:v]scale_cuda=w=480:h=270[pip];[0:v][pip]overlay_cuda=x=1400:y=50" \
       -c:v h264_nvenc -y output.mp4

Command Breakdown

• -hwaccel cuda -hwaccel_output_format cuda: Placed before each input, this forces FFmpeg to decode the video directly into GPU memory (VRAM) as CUDA frames. This prevents slow data transfers between system RAM (CPU) and VRAM (GPU).
• scale_cuda=w=480:h=270[pip]: Scales the second input (the picture-in-picture video) to a resolution of 480x270 pixels using the GPU. The scaled stream is labeled as [pip].
• overlay_cuda=x=1400:y=50: This filter overlays the [pip] stream onto the background stream [0:v] at the specified coordinates (X=1400, Y=50 from the top-left corner). This operation is computed entirely within GPU memory.
• -c:v h264_nvenc: Encodes the final output stream using the NVIDIA hardware H.264 encoder.

Advanced Options

The overlay_cuda filter supports specific parameters to control the layout and timing:

• shortest: If set to 1 (e.g., overlay_cuda=x=10:y=10:shortest=1), the output video will terminate as soon as the shortest input video ends.
• eof_action: Action to take when the secondary input stream ends. Options include repeat (repeats the last frame, default), initialize (displays a blank screen), or pass (continues displaying the main video without the overlay).