FFmpeg Crop Video Using Tracking Coordinates

This article explains how to crop a video using dynamic coordinates from an external tracking file. You will learn how to translate frame-by-frame tracking data into an FFmpeg-compatible format and run the command to output a stabilized or cropped video that follows your target subject.

To crop a video dynamically in FFmpeg, you must use the crop filter. The basic syntax for the crop filter is:

-vf "crop=width:height:x:y"

While width and height usually remain constant, the x and y coordinates must change on a frame-by-frame basis to follow your tracking data. Because FFmpeg cannot natively read raw tracking CSV or TXT files directly inside the filter, you must format your tracking coordinates into an FFmpeg expression or use a script to generate the command.

Method 1: Using FFmpeg Expression Evaluation

FFmpeg’s crop filter evaluates expressions for x and y for every frame. If your tracking coordinates can be represented by mathematical equations or nested conditional statements, you can pass them directly.

For example, if you want the crop box to move based on the frame number n, you can use the if and eq functions:

ffmpeg -i input.mp4 -vf "crop=640:480:'if(eq(n,0),100,if(eq(n,1),105,if(eq(n,2),110,115)))':'if(eq(n,0),200,if(eq(n,1),202,if(eq(n,2),205,210)))'" output.mp4

In this command: * 640:480 defines the width and height of the cropped output. * The expression for x checks the frame number n and assigns 100 for frame 0, 105 for frame 1, 110 for frame 2, and 115 for any subsequent frames. * The expression for y performs a similar check for vertical coordinates.

Method 2: Automating with a Script (Recommended)

Because manual nesting of expressions is highly inefficient for long videos, the standard industry practice is to use a script (such as Python) to parse your tracking file (CSV, TXT, or JSON) and generate the FFmpeg command dynamically.

Suppose you have a tracking file named tracking.csv with the following structure:

frame,x,y
0,120,250
1,122,248
2,125,245

You can use a Python script to build a formatted string of conditional expressions for FFmpeg:

import csv
import subprocess

# Load tracking data
x_expr = ""
y_expr = ""

with open('tracking.csv', 'r') as f:
    reader = csv.DictReader(f)
    for row in reader:
        frame = row['frame']
        x = row['x']
        y = row['y']
        
        # Build nested IF expressions
        x_expr = f"if(eq(n,{frame}),{x},{x_expr if x_expr else x})"
        y_expr = f"if(eq(n,{frame}),{y},{y_expr if y_expr else y})"

# Define crop dimensions
crop_w = 640
crop_h = 480

# Assemble and run the FFmpeg command
command = [
    'ffmpeg', '-i', 'input.mp4',
    '-vf', f"crop={crop_w}:{crop_h}:{x_expr}:{y_expr}",
    '-c:a', 'copy', 'output.mp4'
]

subprocess.run(command)

This script reads each coordinate, generates a nested condition that maps the exact tracking point to its corresponding frame number, and executes FFmpeg to render the cropped video.