Custom Masks with FFmpeg Maskedmerge Filter

The FFmpeg maskedmerge filter blends two video or image streams using a third grayscale video stream as a matte, where white pixels display the overlay, black pixels display the base layer, and intermediate gray levels create varying transparency. This guide details how to generate custom shapes—such as rectangular cutouts, feathered circles, and static graphic assets—directly within FFmpeg's -filter_complex engine to mask and composite video layers without needing third-party editing software.

Understanding maskedmerge Syntax

The maskedmerge filter requires exactly three video streams in a defined order:

  1. First input ([0:v]): The background or base layer.
  2. Second input ([1:v]): The foreground or overlay layer.
  3. Third input ([2:v] or [mask]): The mask layer (white reveals foreground, black preserves background).

All three inputs must share the same resolution and pixel format. Inserting format=yuv420p across all inputs before feeding them to maskedmerge prevents color space mismatch errors.

Method 1: Drawing a Rectangular Mask with drawbox

You can generate a solid mask dynamically using the color source and the drawbox filter. This approach is ideal for picture-in-picture boxes, split screens, or rectangular reveal animations.

ffmpeg -i background.mp4 -i overlay.mp4 -filter_complex \
"[0:v]format=yuv420p[base]; \
 [1:v]format=yuv420p[over]; \
 color=c=black:s=1920x1080[mask_bg]; \
 [mask_bg]drawbox=x=200:y=200:w=800:h=600:color=white:t=fill,format=yuv420p[mask]; \
 [base][over][mask]maskedmerge" \
-c:a copy output.mp4

In this command:

  • color=c=black:s=1920x1080 generates a solid black canvas matching the source dimensions.
  • drawbox=x=200:y=200:w=800:h=600:color=white:t=fill draws a filled white rectangle where the overlay video will appear.

Method 2: Creating a Circular or Feathered Mask with geq

For non-linear shapes like spotlights, circular masks, or feathered gradients, use the geq (generic equation) filter. The equation computes the distance from the center of the frame to determine pixel brightness.

ffmpeg -i background.mp4 -i overlay.mp4 -filter_complex \
"[0:v]format=yuv420p[base]; \
 [1:v]format=yuv420p[over]; \
 nullsrc=s=1920x1080[canvas]; \
 [canvas]geq=lum='if(lte(hypot(X-W/2,Y-H/2),300),255,0)':cb=128:cr=128,format=yuv420p[mask]; \
 [base][over][mask]maskedmerge" \
-c:a copy output.mp4

To create a smooth, feathered edge rather than a hard boundary, replace the step condition with a Gaussian-like or linear falloff:

[canvas]geq=lum='clip(255*(1 - hypot(X-W/2,Y-H/2)/400),0,255)':cb=128:cr=128,format=yuv420p[mask]
  • hypot(X-W/2, Y-H/2) calculates the distance of each coordinate (X, Y) from the center (W/2, H/2).
  • clip(..., 0, 255) clamps the luminance range between complete opacity (255) and complete transparency (0).

Method 3: Using an External Image as a Custom Mask

For intricate shapes, logos, or hand-drawn silhouettes, a static grayscale PNG image can serve as the mask input.

ffmpeg -i background.mp4 -i overlay.mp4 -loop 1 -i mask.png -filter_complex \
"[0:v]format=yuv420p[base]; \
 [1:v]scale=1920:1080,format=yuv420p[over]; \
 [2:v]scale=1920:1080,format=yuv420p[mask]; \
 [base][over][mask]maskedmerge=shortest=1" \
-c:a copy output.mp4

Key considerations for image masks:

  • -loop 1 keeps the static image input active for the full duration of the video.
  • shortest=1 on the maskedmerge filter terminates encoding when the shortest primary video stream finishes, preventing infinite loops.