How to Use FFmpeg Remap Filter for Video Distortion

This article provides a comprehensive guide on using FFmpeg's remap filter to apply arbitrary geometric distortions, lens corrections, and spatial transformations to video files. You will learn the mechanics behind the filter, how coordinate map files are structured, how to generate these maps programmatically, and the exact FFmpeg commands required to process your video.

How the Remap Filter Works

The FFmpeg remap filter rearranges the pixels of an input video based on two separate mapping streams: an X-map and a Y-map.

  • Source Video (Input 0): The video you want to distort.
  • X-Map (Input 1): A grayscale image or video where each pixel's brightness specifies the horizontal (X) source coordinate to sample from.
  • Y-Map (Input 2): A grayscale image or video where each pixel's brightness specifies the vertical (Y) source coordinate to sample from.

Because standard video resolutions exceed 255 pixels, standard 8-bit grayscale cannot represent the full coordinate space. The map files must use 16-bit unsigned grayscale (gray16le), allowing coordinate mapping up to 65,535 × 65,535 pixels.

Step 1: Generating the Map Files

To distort a 1920×1080 video, you must produce two 1920×1080 16-bit grayscale images (or videos) where pixel values represent absolute source coordinates. A pixel value of 0 references the first row/column, 1 references the second, and so on.

You can generate these maps using Python with NumPy and OpenCV. Below is an example that creates a wave/ripple distortion map:

import cv2
import numpy as np

width, height = 1920, 1080

# Create standard coordinate grids
grid_x, grid_y = np.meshgrid(np.arange(width), np.arange(height))

# Apply custom transformation formulas (e.g., horizontal sine wave distortion)
distortion_strength = 30.0
frequency = 0.02
map_x = grid_x + distortion_strength * np.sin(grid_y * frequency)
map_y = grid_y + distortion_strength * np.cos(grid_x * frequency)

# Clip to valid boundaries and cast to 16-bit unsigned integers
map_x = np.clip(map_x, 0, width - 1).astype(np.uint16)
map_y = np.clip(map_y, 0, height - 1).astype(np.uint16)

# Save as 16-bit single-channel PNG files
cv2.imwrite("xmap.png", map_x)
cv2.imwrite("ymap.png", map_y)

Step 2: Running the FFmpeg Remap Command

Once the maps are generated, pass all three inputs into FFmpeg using filter_complex:

ffmpeg -i input.mp4 -i xmap.png -i ymap.png \
  -filter_complex "[0:v][1:v][2:v]remap=interp=bilinear:fill=black" \
  -c:a copy output.mp4

Parameter Breakdown

  • [0:v][1:v][2:v]: Directs the input video stream, the X-map stream, and the Y-map stream into the remap filter in the required order.
  • interp: Specifies the pixel interpolation method:
    • linear or bilinear (default): Smooths mapped pixels, preventing aliasing and pixelated edges.
    • nearest: Faster, but produces aliased/jagged edges.
  • fill: Defines the color used for coordinates that map outside the source image boundaries. Common values are black, white, or transparent.

Creating Animated Distortions

If you want the distortion to change over time, replace the static PNG maps with 16-bit grayscale video streams.

You can render an image sequence of X and Y maps (x_0001.png, y_0001.png, etc.) and feed them directly to FFmpeg:

ffmpeg -i input.mp4 \
  -framerate 30 -i x_%04d.png \
  -framerate 30 -i y_%04d.png \
  -filter_complex "[0:v][1:v][2:v]remap=interp=bilinear" \
  -c:v libx264 -pix_fmt yuv420p output.mp4

Ensure the framerate and duration of the map sequences match the input video to maintain continuous synchronization.