FFmpeg Fish-Eye Correction with Remap Filter

This article explains how to correct fish-eye lens distortion in video files using FFmpeg's remap filter alongside custom coordinate maps. You will learn the mechanics of the remap filter, how to generate custom X and Y displacement maps using Python and OpenCV, and how to execute the final FFmpeg command to render undistorted video.

Understanding the remap Filter

The FFmpeg remap filter transforms an image by mapping source pixels to new destination coordinates. It requires three distinct video or image inputs:

  1. Input 0: The source video to be corrected.
  2. Input 1: The X-map, an image defining the horizontal source coordinate for each destination pixel.
  3. Input 2: The Y-map, an image defining the vertical source coordinate for each destination pixel.

The dimensions of the X and Y maps dictate the resolution of the output video. Pixel values in these maps store coordinate data, typically represented as 16-bit unsigned integers (gray16le) to support pixel dimensions up to 65535×65535.

Step 1: Generating Custom Coordinate Maps

To un-distort fish-eye footage, compute an inverse mapping grid that transforms rectilinear destination coordinates back into distorted source coordinates. A script using Python, NumPy, and OpenCV generates the necessary 16-bit PGM map files based on lens calibration parameters.

import cv2
import numpy as np

# Output dimensions
width = 1920
height = 1080

# Camera matrix (fx, fy: focal lengths; cx, cy: principal point)
K = np.array([[800.0, 0.0, width / 2.0],
              [0.0, 800.0, height / 2.0],
              [0.0, 0.0, 1.0]], dtype=np.float32)

# Fish-eye distortion coefficients (k1, k2, k3, k4)
D = np.array([-0.05, 0.01, -0.005, 0.0], dtype=np.float32)

# Compute identity rectification and new camera matrix
R = np.eye(3, dtype=np.float32)
new_K = cv2.fisheye.estimateNewCameraMatrixForUndistortRectify(K, D, (width, height), np.eye(3), balance=0.0)

# Generate undistortion maps
map_x, map_y = cv2.fisheye.initUndistortRectifyMap(K, D, R, new_K, (width, height), cv2.CV_32FC1)

# Convert float coordinates to 16-bit unsigned integers
map_x_16 = np.clip(np.round(map_x), 0, 65535).astype(np.uint16)
map_y_16 = np.clip(np.round(map_y), 0, 65535).astype(np.uint16)

# Save as 16-bit PGM files
cv2.imwrite("xmap.pgm", map_x_16)
cv2.imwrite("ymap.pgm", map_y_16)

This script outputs xmap.pgm and ymap.pgm, which serve as look-up tables containing the source pixel indices for every pixel in the corrected frame.

Step 2: Applying the Maps in FFmpeg

Run FFmpeg with the source video and the two generated .pgm map files. Use the -lavfi flag to feed all three streams into the remap filter:

ffmpeg -i input.mp4 -i xmap.pgm -i ymap.pgm -lavfi "[0:v][1:v][2:v]remap" -c:v libx264 -crf 18 -c:a copy output.mp4

Map Generation and Command Considerations

  • Interpolation Modes: By default, the remap filter uses bilinear interpolation. You can explicitly set nearest-neighbor, bilinear, or bicubic interpolation using the interp option:

    -lavfi "[0:v][1:v][2:v]remap=interp=bicubic"

  • Pixel Formats: Ensure that map files generated manually or without OpenCV are encoded in gray16le format so FFmpeg reads coordinate data precisely without downsampling artifacts.