Configure FFmpeg find_rect Threshold and Scaling

The FFmpeg find_rect filter enables template matching by searching for a specific rectangular image within video frames. This guide covers how to adjust the detection threshold to control matching sensitivity and configure scaling options via mipmaps and input scaling filters to ensure accurate detection across varying object sizes.

The find_rect Filter Syntax

The basic syntax for applying the find_rect filter in an FFmpeg command line is:

ffmpeg -i input.mp4 -vf "find_rect=object=template.png:threshold=0.8:mipmaps=3" output.mp4
  • object: The file path to the reference image or template being searched for.
  • threshold: The detection threshold value.
  • mipmaps: The number of mipmap scaling levels used for search.

Configuring the Search Threshold

The threshold parameter defines the strictness of the template matching algorithm. It accepts a floating-point value, typically evaluated between 0.0 and 1.0.

  • Higher Threshold (e.g., 0.8 to 0.95): Restricts matches to near-identical pixel regions. Use higher values to eliminate false positives in clean, high-bitrate video streams.
  • Lower Threshold (e.g., 0.3 to 0.5): Loosens match criteria. Use lower values when the source video has compression artifacts, slight color shifts, or noise.
  • Default Value: If omitted, FFmpeg defaults the threshold to 0.5.

Example with a strict threshold:

ffmpeg -i video.mp4 -vf "find_rect=object=logo.png:threshold=0.85" -f null -

Configuring Scaling Options

The find_rect filter handles scale variations using two primary methods: internal mipmaps and external stream pre-scaling.

1. Using Mipmaps for Downscaled Detection

The mipmaps parameter configures multi-scale matching by downsampling the search template and video frames into a mipmap pyramid. Each mipmap level halves the resolution of the search space.

  • Default Value: 3 levels (searches at 100%, 50%, and 25% scale).
  • Increasing Mipmaps: Setting mipmaps=4 or mipmaps=5 expands the search to smaller object sizes (down to 12.5% and 6.25%, respectively).
  • Disabling Multi-Scale: Setting mipmaps=1 forces the filter to search only at the exact 1:1 pixel scale of the reference image, which significantly improves processing speed if the object size is static.

Example with expanded multi-scale searching:

ffmpeg -i video.mp4 -vf "find_rect=object=target.png:threshold=0.7:mipmaps=5" -f null -

2. Pre-Scaling with the scale Filter

The native mipmaps option only scales downward in powers of two. If the target object in the video appears at an arbitrary or larger scale than the template image, normalize the template or the video stream using the scale filter before feeding it into find_rect.

To resize the video prior to running the detection:

ffmpeg -i video.mp4 -vf "scale=1280:720,find_rect=object=template.png:threshold=0.75:mipmaps=2" output.mp4

Optimizing Search Performance

Multi-scale searching with a low threshold is computationally intensive. To optimize execution speed:

  1. Restrict Search Area: Use xmin, ymin, xmax, and ymax to confine the search to regions where the object is expected to appear.
  2. Limit Mipmaps: Keep mipmaps as low as possible if the object dimensions do not change across frames.
  3. Tune Sequentially: Begin testing with threshold=0.5 and mipmaps=1, then adjust the threshold to isolate the object before increasing mipmaps to account for size variations.