Configure FFmpeg SR Filter Scale Factor and Backend

This guide explains how to configure the scaling factor and execution backend—native versus TensorFlow—in FFmpeg's sr (Super-Resolution) video filter. The sr filter applies deep learning models to upscale video frames, requiring you to define the computational engine via the backend option and align the scaling factor to match the specific model file being loaded.

Understanding the Parameters

The sr filter relies on three primary parameters to control scaling and processing:

  • dnn_backend: Defines the inference engine. Accepted values are native and tensorflow.
  • scale_factor: Sets the resolution multiplier (e.g., 2, 3, or 4). This value must match the scale factor the selected model was trained to output.
  • model: Specifies the path to the trained model file.

Backend Comparison: Native vs. TensorFlow

  • native: FFmpeg's built-in deep neural network (DNN) engine. It requires no external dependencies and runs models converted to FFmpeg's native .model format. It is lightweight and works out of the box in most builds.
  • tensorflow: Uses the external TensorFlow C API (libtensorflow). It runs standard TensorFlow frozen graph (.pb) files. To use this backend, FFmpeg must be compiled with the --enable-libtensorflow configuration flag.

Configuring the Native Backend

To use the native backend, supply a native format model file and set dnn_backend=native.

ffmpeg -i input.mp4 -vf "sr=dnn_backend=native:scale_factor=2:model=espcn_x2.model" -c:a copy output.mp4

In this command:

  • dnn_backend=native engages the internal engine.
  • scale_factor=2 doubles the width and height of the video.
  • espcn_x2.model is the path to the model trained specifically for a 2x upscale.

Configuring the TensorFlow Backend

To use the TensorFlow backend, supply a frozen TensorFlow graph (.pb) and set dnn_backend=tensorflow.

ffmpeg -i input.mp4 -vf "sr=dnn_backend=tensorflow:scale_factor=4:model=espcn_x4.pb" -c:a copy output.mp4

In this command:

  • dnn_backend=tensorflow routes execution through the TensorFlow C library.
  • scale_factor=4 quadruples the width and height of the input frames.
  • espcn_x4.pb is the corresponding 4x model file.

Key Requirements and Constraints

  1. Scale Matching: The scale_factor argument does not dynamically resize the model's output layer; it informs FFmpeg what output dimensions to expect from the model. Passing a scale factor that conflicts with the model architecture will cause the filter to fail.
  2. Color Space: The sr filter typically processes the luma (Y) channel. Ensure your pipeline converts pixel formats (such as yuv420p) appropriately if your model requires a specific input format.