How to Use FFmpeg find_rect for Image Search

This guide explains how to use FFmpeg's built-in find_rect video filter to locate a static template image within a video stream. It covers the required syntax, configuration parameters, methods for logging detection coordinates via frame metadata, and visual verification techniques using drawing filters.

Overview of the find_rect Filter

The find_rect filter searches video frames for a specified template image using normalized cross-correlation. When a match exceeds the defined similarity threshold, the filter writes the coordinates and dimensions of the match directly to frame metadata.

The filter assigns the following metadata keys to matched frames:

  • lavfi.rect.x: The X coordinate of the top-left corner of the match.
  • lavfi.rect.y: The Y coordinate of the top-left corner of the match.
  • lavfi.rect.w: The width of the matched object.
  • lavfi.rect.h: The height of the matched object.
  • lavfi.rect.score: The correlation score of the match.

Basic Syntax and Options

The core syntax requires specifying the path to the template image file using the object option:

ffmpeg -i input.mp4 -vf "find_rect=object=template.png:threshold=0.8" -f null -

Key Parameters

  • object: The path to the image file containing the pattern to find. The format is typically PNG or BMP.
  • threshold: A floating-point number between 0.0 and 1.0 (default is 0.5). A higher threshold requires a closer visual match.
  • mipmaps: Sets the number of mipmap levels used for the search (default is 3). Higher values allow searching across scales but decrease speed.
  • xmin, ymin, xmax, ymax: Constrains the search area to a specific rectangular subregion within each frame to optimize processing time.

Logging Coordinates to a Text File

To extract the exact timestamps and coordinates where the template appears, chain find_rect with the metadata filter to export detection logs:

ffmpeg -i input.mp4 -vf "find_rect=object=template.png:threshold=0.85,metadata=print:key=lavfi.rect:file=detections.txt" -f null -

The output file detections.txt will record frame timestamps and metadata tags containing lavfi.rect.x and lavfi.rect.y whenever a match is identified.

Highlighting Matches Visually

To visually verify matches in the output video, pass the metadata variables directly to the drawbox filter:

ffmpeg -i input.mp4 -vf "find_rect=object=template.png:threshold=0.85,drawbox=x='metadata:lavfi.rect.x':y='metadata:lavfi.rect.y':w='metadata:lavfi.rect.w':h='metadata:lavfi.rect.h':color=red@0.8:t=2" -c:a copy output.mp4

This draws a red bounding box around the detected object for every frame where the correlation threshold is satisfied.

Performance Optimization

  1. Resolution Matching: Ensure template.png matches the scale and color space of the target within the video. find_rect performs best when the template is an exact pixel-for-pixel crop.
  2. Restrict Search Area: If the object only appears in a specific quadrant (e.g., a channel logo), define bounding bounds using xmin, ymin, xmax, and ymax to reduce CPU usage.
  3. Format Conversion: Ensure the video stream and the template share a compatible pixel format (such as yuv420p or rgb24) by placing a format filter prior to find_rect if decoding mismatches occur.