How to Use FFmpeg Remap Filter for Video Distortion

The remap filter in FFmpeg is a powerful tool that allows you to rearrange pixels in a video frame to create custom geometric distortions, such as fisheye effects, perspective corrections, lens distortions, or panoramic projections. This article provides a straightforward guide on how the remap filter works, how to generate the required coordinate map files using Python, and how to execute the FFmpeg command to transform your videos.

Understanding the FFmpeg Remap Filter

The remap filter works by taking three inputs: 1. The source video you want to distort. 2. An X-map (a grayscale image or video representing the source X-coordinates). 3. A Y-map (a grayscale image or video representing the source Y-coordinates).

For every pixel in the destination video at coordinate (x_dst, y_dst), the filter looks at the pixel values in the X-map and Y-map at that same coordinate. These values tell the filter which pixel (x_src, y_src) from the input video should be copied to the destination.

Because standard 8-bit images only support values from 0 to 255, FFmpeg uses 16-bit grayscale PGMs (Portable Graymap) for the maps. This allows coordinate mapping for resolutions up to 65,535 x 65,535 pixels.

Step 1: Generating the Map Files

To apply a custom distortion, you must first generate the X and Y map files. You can do this easily using a Python script with the numpy and cv2 (OpenCV) libraries.

Below is a Python script that generates maps for a sinusoidal wave distortion on a 1920x1080 video:

import numpy as np
import cv2

width, height = 1920, 1080

# Create grids for destination coordinates
map_x, map_y = np.meshgrid(np.arange(width), np.arange(height))

# Define custom geometric distortion (e.g., a wave effect)
# We calculate source coordinates based on destination coordinates
source_x = map_x + 20.0 * np.sin(map_y / 30.0)
source_y = map_y + 20.0 * np.cos(map_x / 30.0)

# Clip coordinates to prevent out-of-bounds errors
source_x = np.clip(source_x, 0, width - 1).astype(np.uint16)
source_y = np.clip(source_y, 0, height - 1).astype(np.uint16)

# Save as 16-bit PGM files (P5 binary format)
cv2.imwrite('map_x.pgm', source_x)
cv2.imwrite('map_y.pgm', source_y)

print("Map files generated successfully.")

This script outputs two 16-bit files: map_x.pgm and map_y.pgm.

Step 2: Applying the Maps in FFmpeg

Once you have your map files, you can use the FFmpeg remap filter. The command structure passes the video as the first input, followed by the X-map and Y-map.

Run the following command in your terminal:

ffmpeg -i input.mp4 -i map_x.pgm -i map_y.pgm -filter_complex "[0:v][1:v][2:v]remap" output.mp4

Command Breakdown:

By modifying the mathematical formulas in the Python script, you can achieve any geometric transformation, including custom 360-degree projections, spherical mapping, or lens correction.