Configure FFmpeg SR Filter Scale and Backend

This guide explains how to configure the scale factor and choose between the TensorFlow and native backends in the FFmpeg super-resolution (sr) video filter. You will learn the exact syntax, parameter configurations, and prerequisites needed to upscale videos using deep learning models within FFmpeg.

Understanding the FFmpeg SR Filter

The sr (super-resolution) filter in FFmpeg uses deep learning models to upscale images and videos. To use this filter, you must define the hardware/software backend, specify the scale factor, and provide a pre-trained model compatible with your chosen backend.

The filter accepts three primary arguments:

  • dnn_backend: The engine used to run the model (native or tensorflow).
  • scale_factor: The upscaling ratio (typically 2, 3, or 4).
  • model: The path to the pre-trained model file.

Configuring the DNN Backend

FFmpeg supports two backends for the super-resolution filter. Your choice dictates how the model file is generated and how FFmpeg must be compiled.

1. The Native Backend (native)

The native backend uses FFmpeg's built-in deep learning engine. It does not require external libraries, making it highly portable.

  • Prerequisites: Works with any standard FFmpeg installation.

  • Model Format: Requires a .model file exported specifically for FFmpeg's native format (often converted from TensorFlow using FFmpeg's python scripts).

  • Example Command:

    ffmpeg -i input.mp4 -vf sr=dnn_backend=native:scale_factor=2:model=espcn_x2.model output.mp4

2. The TensorFlow Backend (tensorflow)

The TensorFlow backend utilizes the official TensorFlow C API. It is generally faster and supports more complex network architectures than the native backend.

  • Prerequisites: FFmpeg must be compiled with the --enable-libtensorflow flag, and the TensorFlow C library must be installed on your system.

  • Model Format: Requires a frozen TensorFlow GraphDef file (.pb).

  • Example Command:

    ffmpeg -i input.mp4 -vf sr=dnn_backend=tensorflow:scale_factor=2:model=espcn_x2.pb output.mp4

Configuring the Scale Factor

The scale_factor parameter defines the target resolution multiplier.

scale_factor=2

Important Scale Factor Rules:

  1. Model Alignment: The scale_factor value must match the scale factor the model was trained on. If you use a model trained for 2x upscaling (e.g., espcn_x2), you must set scale_factor=2. Setting it to any other number will result in an execution error or distorted output.
  2. Supported Scales: Common values are 2, 3, and 4.

Complete Implementation Examples

Example 1: 2x Upscale using the Native Backend

If you have a native format model named srcnn_x2.model and want to upscale a video by 2x:

ffmpeg -i input_1080p.mp4 -vf sr=dnn_backend=native:scale_factor=2:model=srcnn_x2.model output_4k.mp4

Example 2: 4x Upscale using the TensorFlow Backend

If you have a compiled version of FFmpeg with TensorFlow support and a 4x TensorFlow model named espcn_x4.pb:

ffmpeg -i input_720p.mp4 -vf sr=dnn_backend=tensorflow:scale_factor=4:model=espcn_x4.pb output_4k.mp4