Upscale Video with FFmpeg Super Resolution Filter

This guide explains how to use the Super Resolution (sr) filter in FFmpeg to upscale videos using deep learning models. You will learn how to verify your FFmpeg installation for deep neural network (DNN) support, obtain the necessary pre-trained models (such as SRCNN or ESPCN), and execute the correct command-line syntax to significantly enhance your video quality.

Prerequisites

To use the sr filter, your version of FFmpeg must be compiled with DNN backend support. The filter supports three backends: Native, TensorFlow, and OpenVINO.

You can check if your FFmpeg build supports these backends by running:

ffmpeg -filters | grep sr

If the sr filter is listed, your build is ready. For optimal performance, it is highly recommended to use a build compiled with --enable-libtensorflow or --enable-libopenvino.

Step 1: Obtain the Super Resolution Models

The sr filter requires a pre-trained neural network model file (usually in .pb format for TensorFlow or .onnx for native/other backends). The two most common architectures used with FFmpeg are:

• SRCNN (Super-Resolution Convolutional Neural Network): Highly accurate but computationally expensive.
• ESPCN (Efficient Sub-Pixel Convolutional Neural Network): Faster and more efficient for real-time processing.

You can download pre-compiled model files from trusted open-source repositories or train your own using TensorFlow and export them to the .pb format compatible with FFmpeg.

Step 2: The FFmpeg SR Command Syntax

The basic syntax for the sr filter lies within the video filter (-vf) flag. The filter requires you to specify the DNN backend and the path to your model file.

ffmpeg -i input.mp4 -vf "sr=dnn_backend=tensorflow:model=espcn.pb" output.mp4

Parameter Breakdown

• dnn_backend: Defines the backend to use. Options are native, tensorflow, or openvino.

• model: The relative or absolute path to your pre-trained model file (e.g., espcn.pb or srcnn.pb).

• scale_factor (optional): Specifies the upscale factor (usually 2, 3, or 4 depending on what the model was trained on). If the model configuration does not set this automatically, you can define it manually:

  -vf "sr=dnn_backend=tensorflow:model=espcn.pb:scale_factor=2"

Example: Upscaling 1080p to 4K using ESPCN

If you have a 1080p video and want to upscale it to 4K (a scale factor of 2) using the TensorFlow backend and an ESPCN model, use the following command:

ffmpeg -i input_1080p.mp4 -vf "sr=dnn_backend=tensorflow:model=espcn_x2.pb" -c:v libx264 -crf 17 -c:a copy output_4k.mp4

Tips for Best Performance

• Hardware Acceleration: Deep learning upscaling is CPU-intensive. If your FFmpeg is compiled with OpenVINO (dnn_backend=openvino), you can leverage Intel GPUs to dramatically speed up the upscaling process.
• Scale First, Post-Process Later: If you need to perform other filtering actions (like color correction or denoising), place the sr filter at the end of your filter chain to avoid processing extra pixels unnecessarily.