FFmpeg OpenCL Hardware Accelerated Video Overlay

This article provides a straightforward guide on how to use the overlay_opencl filter in FFmpeg to perform hardware-accelerated video overlays. You will learn how to initialize an OpenCL device, upload video frames to GPU memory, apply the overlay filter, and retrieve the processed video, significantly boosting processing speeds compared to traditional CPU-based overlay filters.

Prerequisites

To use OpenCL filters, your FFmpeg build must be compiled with OpenCL support (verify this by running ffmpeg -protocols and looking for opencl, or checking if --enable-opencl was used during compilation). You also need a compatible GPU with up-to-date OpenCL drivers installed on your system.

The OpenCL Processing Pipeline

Because overlay_opencl runs on the GPU, both the main video and the overlay input must be transferred from system memory (CPU) to hardware memory (GPU) before applying the filter. After processing, the output must be downloaded back to system memory for software encoding, or passed directly to a hardware encoder.

Command Example: CPU to GPU to CPU

The following command demonstrates how to take two standard video inputs, upload them to the GPU, overlay one onto the other using OpenCL, download the result, and encode it using a standard software encoder (x264):

ffmpeg -init_hw_device opencl=gpu -filter_hw_device gpu \
-i main_input.mp4 -i overlay_input.mp4 \
-filter_complex \
"[0:v]format=nv12,hwupload[main_gpu]; \
 [1:v]format=nv12,hwupload[overlay_gpu]; \
 [main_gpu][overlay_gpu]overlay_opencl=x=100:y=50[out_gpu]; \
 [out_gpu]hwdownload,format=nv12" \
-c:v libx264 -c:a copy output.mp4

Command Breakdown

-init_hw_device opencl=gpu: Initializes the OpenCL hardware device and aliases it as gpu.
-filter_hw_device gpu: Instructs the filter graph to use the initialized gpu device for hardware filters.
format=nv12,hwupload: Converts the pixel format of the input to nv12 (a widely supported hardware format) and uploads the frames to the GPU. This is done for both the main video ([0:v]) and the overlay video ([1:v]).
overlay_opencl=x=100:y=50: The hardware-accelerated overlay filter. It positions the top-left corner of the overlay video at coordinates X=100, Y=50 on the main video.
hwdownload,format=nv12: Downloads the processed frame back to system memory and defines the output pixel format so the software encoder can read it.

Parameters for `overlay_opencl`

The overlay_opencl filter supports the following main parameters:

x: The horizontal position of the overlay video (defaults to 0).
y: The vertical position of the overlay video (defaults to 0).

You can use expressions for these parameters, such as main_w (main width), main_h (main height), overlay_w (overlay width), and overlay_h (overlay height).

For example, to center the overlay:

overlay_opencl=x=(main_w-overlay_w)/2:y=(main_h-overlay_h)/2

Full Hardware Pipeline (Advanced)

If you want to avoid downloading the video back to the CPU, you can pair the OpenCL overlay directly with hardware-accelerated decoding and encoding (such as VAAPI, QSV, or NVENC) depending on your GPU manufacturer. This keeps the entire pipeline on the GPU, maximizing performance.