Correct Fish-Eye Distortion with FFmpeg Remap Filter

Correcting fish-eye lens distortion in video files can be precisely achieved using FFmpeg’s powerful remap filter. This article explains how to use the remap filter to fix fish-eye distortion by generating and applying custom X and Y coordinate maps. You will learn the mathematical concepts behind coordinate mapping, how to generate these map files using a Python script, and how to execute the FFmpeg command to produce a rectilinearly corrected video.

How the FFmpeg Remap Filter Works

The remap filter rearranges pixels from a source video to a destination video using two coordinate map files: 1. X-map: Defines the source \(X\) coordinate for every destination pixel \((x, y)\). 2. Y-map: Defines the source \(Y\) coordinate for every destination pixel \((x, y)\).

These map files are passed to FFmpeg as secondary video or image inputs. FFmpeg supports 16-bit grayscale PGMs (Portable Graymap) for these maps, which allows for sub-pixel accuracy and supports high-resolution videos (up to \(65535 \times 65535\) pixels).

Step 1: Generate the Custom Maps

To correct fish-eye distortion, we must map the distorted coordinates of the source image back to flat, rectilinear coordinates. Below is a Python script that uses numpy to calculate these maps and write them as 16-bit binary PGMs.

This script uses the division model for radial distortion: \[r_d = \frac{r_u}{1 + k \cdot r_u^2}\]

import numpy as np

def generate_fisheye_maps(width, height, k):
    # Create coordinate grid for destination image
    x_indices = np.arange(width, dtype=np.float32)
    y_indices = np.arange(height, dtype=np.float32)
    x_grid, y_grid = np.meshgrid(x_indices, y_indices)

    # Normalize coordinates to center (0, 0)
    cx, cy = width / 2.0, height / 2.0
    x_normalized = (x_grid - cx) / cx
    y_normalized = (y_grid - cy) / cy

    # Calculate radial distance
    r_u = np.sqrt(x_normalized**2 + y_normalized**2)

    # Apply radial distortion model (division model)
    # k > 0 corrects barrel (fish-eye) distortion
    distortion_factor = 1.0 + k * (r_u**2)
    
    x_src_normalized = x_normalized * distortion_factor
    y_src_normalized = y_normalized * distortion_factor

    # Convert back to absolute pixel coordinates
    map_x = (x_src_normalized * cx) + cx
    map_y = (y_src_normalized * cy) + cy

    # Clip coordinates to prevent out-of-bound errors
    map_x = np.clip(map_x, 0, width - 1)
    map_y = np.clip(map_y, 0, height - 1)

    return map_x, map_y

def save_pgm_16bit(filename, array):
    height, width = array.shape
    # Scale coordinates to 16-bit integer representation
    # FFmpeg remap expects raw pixel values to match source coordinate values
    data = array.astype(np.uint16)
    
    # Bytes must be written in big-endian format for PGM P5
    if data.dtype.byteorder == '<' or (data.dtype.byteorder == '=' and np.dtype('u2').byteorder == '<'):
        data = data.byteswap()

    with open(filename, 'wb') as f:
        # PGM Header: P5 (Binary), Width, Height, Max Value (65535)
        header = f"P5\n{width} {height}\n65535\n"
        f.write(header.encode('ascii'))
        f.write(data.tobytes())

# Configuration
width = 1920
height = 1080
k = 0.22  # Distortion coefficient (adjust based on lens severity)

map_x, map_y = generate_fisheye_maps(width, height, k)
save_pgm_16bit('map_x.pgm', map_x)
save_pgm_16bit('map_y.pgm', map_y)
print("Map files generated successfully.")

Adjust the coefficient k in the script depending on your camera lens. A higher positive value corrects stronger barrel distortion, while a negative value will introduce barrel distortion.

Step 2: Run the FFmpeg Command

Once you have generated the map_x.pgm and map_y.pgm files, you can apply them to your video using FFmpeg.

Use the following command to map the source video frames using your custom coordinate maps:

ffmpeg -i input.mp4 -i map_x.pgm -i map_y.pgm -filter_complex "[0:v][1:v][2:v]remap[out]" -map "[out]" -map 0:a? -c:v libx264 -crf 18 -c:a copy output.mp4

Command Breakdown: