Search Video for Template Image Using FFmpeg find_rect

This article explains how to use the FFmpeg find_rect filter to search for and locate a specific template image within a video stream. You will learn how to prepare your template image, construct the FFmpeg command, and utilize the resulting coordinate metadata to identify where and when the object appears in your video.

Step 1: Prepare the Template Image

The find_rect filter requires the template image (the object you are searching for) to be in a grayscale format, specifically a PGM (Portable Graymap) file. If your template image is in PNG or JPEG format, you must convert it to grayscale PGM first.

You can perform this conversion directly using FFmpeg:

ffmpeg -i template.png -vf format=gray template.pgm

Step 2: Basic Syntax of the find_rect Filter

The find_rect filter compares the template image against every frame of the video. The basic parameters for the filter are:

• object: The path to your prepared .pgm template image.
• threshold: The detection threshold (ranging from 0.0 to 1.0). A lower value makes the search stricter, while a higher value is more lenient. The default is 0.5.
• mipmaps: The number of mipmaps used for multi-scale matching. The default is 3.

Here is the basic filter syntax:

find_rect=object=template.pgm:threshold=0.5

Step 3: Visualizing the Search Results

By default, the find_rect filter does not draw a box on the video; it only generates metadata coordinates for the detected object. To visually verify that the filter is finding your template, you can chain it with the drawbox filter.

The find_rect filter outputs the coordinates to the following metadata keys: * lavfi.rect.x * lavfi.rect.y * lavfi.rect.w (width of the template) * lavfi.rect.h (height of the template)

Use this command to render a red box around the detected template image in the output video:

ffmpeg -i input.mp4 -vf "find_rect=object=template.pgm:threshold=0.5,drawbox=x='val(lavfi.rect.x)':y='val(lavfi.rect.y)':w='val(lavfi.rect.w)':h='val(lavfi.rect.h)':color=red" output.mp4

Step 4: Extracting Coordinate Metadata

If you want to search the video and export the timestamps and exact coordinates of the template without rendering a new video file, you can output the metadata to your console or log files.

Run the filter with the showinfo filter and direct the output to a text file. This avoids the overhead of re-encoding:

ffmpeg -i input.mp4 -vf "find_rect=object=template.pgm:threshold=0.5,showinfo" -f null - > detection_log.txt 2>&1

Open the resulting detection_log.txt file. When the template is successfully found, you will see log entries containing the metadata values:

lavfi.rect.x=120
lavfi.rect.y=340
lavfi.rect.w=50
lavfi.rect.h=50