Generate X and Y Maps for FFmpeg Remap Filter

The FFmpeg remap filter allows you to perform custom spatial transformations and geometric distortions on video by mapping source coordinates to destination coordinates. This article covers how coordinate mapping works in FFmpeg, provides a Python script using NumPy to generate the required binary X and Y map files, and demonstrates the exact FFmpeg command required to process your video with these maps.

Understanding Map File Requirements

The remap filter requires three inputs: the source video, an X-coordinate map, and a Y-coordinate map.

  • Dimensions: Both map files must have the exact resolution (width and height) of the intended output video.
  • Format: The map files are typically raw 16-bit uncompressed grayscale video streams (gray16le).
  • Values: Each pixel in the X-map stores the source horizontal pixel index (ranging from 0 to source_width - 1). Each pixel in the Y-map stores the source vertical pixel index (ranging from 0 to source_height - 1). Values are stored as little-endian unsigned 16-bit integers (uint16).

Generating the Maps with Python and NumPy

Using Python with NumPy is the most efficient way to generate the required gray16le binary files. The following script creates an output resolution of 1920x1080 and applies a horizontal flip as a practical example:

import numpy as np

# Output video dimensions
out_width = 1920
out_height = 1080

# Source video dimensions
src_width = 1920
src_height = 1080

# Create coordinate grids
# x_indices will shape (1080, 1920)
# y_indices will shape (1080, 1920)
x_indices, y_indices = np.meshgrid(np.arange(out_width), np.arange(out_height))

# Define your transformation logic:
# Example: Horizontal mirror (flip across the X-axis)
x_map = (src_width - 1) - x_indices
y_map = y_indices

# Ensure values stay within valid input ranges
x_map = np.clip(x_map, 0, src_width - 1).astype(np.uint16)
y_map = np.clip(y_map, 0, src_height - 1).astype(np.uint16)

# Save the arrays as raw binary files
x_map.tofile("xmap.raw")
y_map.tofile("ymap.raw")

print("Successfully generated xmap.raw and ymap.raw")

Applying Custom Transformations

To generate arbitrary geometric distortions (such as lens correction, fisheye, polar unwarping, or perspective shifts), modify the x_map and y_map formulas:

  • Identity Mapping (No change): Set x_map = x_indices and y_map = y_indices.
  • Cropping / Offset: Add an integer offset to shift the frame (e.g., x_map = x_indices + 100).
  • Non-linear Functions: Use standard trigonometric or polynomial equations to calculate the source coordinates for every destination pixel (x, y).

Running FFmpeg with the Remap Filter

Once xmap.raw and ymap.raw are created, pass them into FFmpeg as raw video inputs alongside your source video:

ffmpeg -i input.mp4 \
  -f rawvideo -pix_fmt gray16le -s 1920x1080 -i xmap.raw \
  -f rawvideo -pix_fmt gray16le -s 1920x1080 -i ymap.raw \
  -filter_complex "[0:v][1:v][2:v]remap" \
  -c:a copy output.mp4

Technical Considerations

  • Interpolation: The remap filter defaults to nearest-neighbor interpolation. For smoother transformations, specify bilinear interpolation using -filter_complex "[0:v][1:v][2:v]remap=interp=bilinear".
  • Out-of-Bounds Handling: If your transformation points to coordinates outside the source frame dimensions, FFmpeg will clamp the edges or output black pixels depending on the fill parameter.
  • File Size: Raw 16-bit maps take up significant disk space (width × height × 2 bytes per frame). For static transformations, generating a single-frame raw file is sufficient, as FFmpeg will loop or hold the static map across the entire duration of the video.