Upscale Video in FFmpeg Using the SR Filter

FFmpeg includes a dedicated sr (Super Resolution) video filter that applies deep learning models to upscale video frames while preserving sharper details than traditional bicubic or lanczos scaling. This guide provides a direct walkthrough on setting up the necessary model files, configuring the deep learning backends, and executing FFmpeg commands using the sr filter to enhance video resolution.

Prerequisites and Backend Requirements

The sr filter relies on FFmpeg's Deep Neural Network (DNN) module. To use it, your FFmpeg build must be compiled with the appropriate backend:

  • TensorFlow: Requires compiling FFmpeg with --enable-libtensorflow.
  • OpenVINO: Requires compiling FFmpeg with --enable-libopenvino.
  • Native: Built into FFmpeg by default, but only supports specific model architectures exported into FFmpeg's custom .model binary format.

Verify your build's enabled filters and libraries by running:

ffmpeg -filters | grep sr

Obtaining and Preparing Deep Learning Models

The sr filter commonly works with architectures such as ESPCN (Efficient Sub-Pixel Convolutional Neural Network) or SRCNN.

For the TensorFlow backend, you need a frozen graph file (.pb) trained for super-resolution:

  1. Obtain or train an ESPCN model using TensorFlow/Keras.
  2. Export the trained weights into a frozen graph (.pb) format with defined input and output nodes (commonly named x and y).

For the Native backend, FFmpeg provides Python tools in its source repository (tools/python/convert_header.py or model conversion scripts) to convert trained checkpoints into native binary .model files.

Basic Syntax of the SR Filter

The sr filter accepts several key parameters:

  • dnn_backend: The inference engine (native, tensorflow, or openvino).
  • model: The path to the model file (.pb or .model).
  • scale_factor: The magnification factor (typically 2, 3, or 4), which must match the scale factor the model was trained on.
  • input: (TensorFlow only) The name of the model's input layer.
  • output: (TensorFlow only) The name of the model's output layer.

Command Examples

Example 1: Upscaling with the TensorFlow Backend

If you have a 2x ESPCN model saved as espcn_2x.pb:

ffmpeg -i input.mp4 -vf "sr=dnn_backend=tensorflow:model=espcn_2x.pb:scale_factor=2:input=x:output=y" -c:v libx264 -crf 18 -c:a copy output_2x.mp4

Example 2: Upscaling with the Native Backend

If using a native FFmpeg .model file:

ffmpeg -i input.mp4 -vf "sr=dnn_backend=native:model=espcn_2x.model:scale_factor=2" -c:v libx264 -crf 18 -c:a copy output_2x.mp4

Example 3: Handling Color Spaces

Super-resolution models often operate only on the luminance channel (Y) in YUV color spaces or require RGB inputs. If the model fails or produces color artifacts, explicitly convert the pixel format before and after applying the sr filter:

ffmpeg -i input.mp4 -vf "format=rgb24,sr=dnn_backend=tensorflow:model=model_rgb.pb:scale_factor=2,format=yuv420p" -c:v libx264 -crf 18 -c:a copy output_upscaled.mp4

Performance Considerations

Deep learning upscaling is computationally demanding compared to standard algorithmic scalers. Processing speeds depend heavily on your CPU or GPU capabilities and frame dimensions. For processing high-definition inputs, ensure hardware acceleration is configured within your TensorFlow or OpenVINO dynamic libraries to optimize throughput.