Custom Video Restoration with FFmpeg and Vapoursynth

This article explains how to implement custom video restoration filters using FFmpeg’s powerful filter_complex engine and the Python-based Vapoursynth framework. You will learn how to chain built-in FFmpeg filters for complex video cleaning pipelines, write Vapoursynth scripts for advanced, frame-by-frame Python-based restoration, and seamlessly integrate Vapoursynth outputs directly into FFmpeg for final encoding.

Custom Restoration Using FFmpeg filter_complex

FFmpeg’s -filter_complex option allows you to construct a directed acyclic graph of video filters. This is ideal for restoration tasks where you need to combine multiple processing steps—such as denoising, deinterlacing, and sharpening—in a specific order.

To perform a standard restoration pipeline that denoises a video using Non-Local Means (nlmeans) and then sharpens the output using the unsharp filter, use the following command structure:

ffmpeg -i input.mp4 -filter_complex \
"[0:v]nlmeans=s=1.0:p=7:pc=0[denoised]; \
 [denoised]unsharp=lx=5:ly=5:la=0.8:cx=5:cy=5:ca=0.0[restored]" \
-map "[restored]" -map 0:a? -c:v libx264 -preset slow -crf 18 -c:a copy output.mp4

How the Filter Graph Works

  1. [0:v]: Takes the video stream from the first input file (input.mp4).
  2. nlmeans: Applies a high-quality spatial denoising algorithm. The s=1.0 parameter controls the denoising strength.
  3. [denoised]: Assigns a temporary label to the output of the denoiser.
  4. unsharp: Takes the [denoised] stream and applies a sharpening mask to restore lost edge details.
  5. -map "[restored]": Tells FFmpeg to use the final processed stream for the output file.

Custom Restoration Using Vapoursynth

Vapoursynth is a modern video processing framework that utilizes Python scripting. It offers superior precision, support for 16-bit/float color depths, and access to highly specialized, community-created restoration plugins (such as Temporal Degrain, BM3D, and Waifu2x).

To use Vapoursynth with FFmpeg, you write a Python script (.vpy) to handle the restoration, and then pipe the raw frames to FFmpeg for encoding.

1. Write the Vapoursynth Script (restoration.vpy)

Create a text file named restoration.vpy and add the following Python code. This script loads a video, converts it to a high bit-depth format, applies a standard convolution filter to smooth out noise, and outputs the result.

import vapoursynth as vs

# Initialize the Vapoursynth core
core = vs.core

# Load the source video (requires the ffms2 plugin)
video = core.ffms2.Source(source="input.mp4")

# Convert to 16-bit integer for high-precision processing
video = core.resize.Bicubic(clip=video, format=vs.YUV420P16)

# Apply a custom 3x3 convolution matrix to smooth out high-frequency noise
# This acts as a customizable low-pass filter
kernel = [1, 2, 1, 
          2, 4, 2, 
          1, 2, 1]
restored_video = core.std.Convolution(clip=video, matrix=kernel)

# Set the final output stream
restored_video.set_output()

2. Pipe Vapoursynth into FFmpeg

Because Vapoursynth outputs uncompressed video, you must pipe its output using the vspipe command-line utility into FFmpeg for compression and encoding.

Run the following command in your terminal:

vspipe --y4m restoration.vpy - | ffmpeg -i - -c:v libx264 -preset slow -crf 18 output.mp4

How the Pipe Works

Choosing Between the Two Methods