FFmpeg Overlay Filter with Different Frame Rates

Overlaying two video streams with different frame rates in FFmpeg can lead to synchronization issues, dropped frames, or stuttering in the output file. This article provides a straightforward guide on how to properly handle varying frame rates using FFmpeg’s overlay filter. You will learn how to synchronize input streams using the fps filter and configure overlay parameters to ensure a smooth, professional, and perfectly timed video output.

The Problem with Mismatched Frame Rates

When you overlay a video (the “overlay” input) onto another (the “main” input) with a different frame rate, FFmpeg has to reconcile the timing difference. By default, the overlay filter synchronizes frames based on the timestamps of the first input. If the overlay stream has a lower or higher frame rate than the main stream, FFmpeg may drop frames or repeat them incorrectly, causing the overlay to appear choppy, laggy, or out of sync.

The Solution: Normalize Frame Rates Using the fps Filter

The most reliable way to handle different frame rates is to normalize them to a matching rate before they reach the overlay filter. This is achieved using FFmpeg’s -filter_complex flag and the fps video filter.

Here is the standard command to match the frame rates of both inputs and overlay them:

ffmpeg -i main_60fps.mp4 -i overlay_24fps.mp4 -filter_complex \
"[0:v]fps=60[main]; \
 [1:v]fps=60[over]; \
 [main][over]overlay=x=10:y=10:shortest=1" \
-c:a copy output.mp4

How the Command Works

1. [0:v]fps=60[main]: This takes the video stream of the first input (main_60fps.mp4) and ensures it is set to 60 frames per second, labeling the output stream as [main].
2. [1:v]fps=60[over]: This takes the video stream of the second input (overlay_24fps.mp4) and upsamples its frame rate to 60 frames per second, labeling the output stream as [over]. FFmpeg smoothly duplicates frames of the 24fps video to match the 60fps target.
3. [main][over]overlay=x=10:y=10:shortest=1: This overlays the newly synchronized [over] stream on top of the [main] stream at the coordinates X=10, Y=10.

Key Parameters for Smooth Overlay Handling

When working with overlays of different lengths and frame rates, these additional parameters inside the overlay filter are highly useful:

• shortest=1: If the overlay video is shorter than the main video, this parameter forces the output to stop encoding when the shortest input ends. Without this, the video may freeze on the last frame of the overlay.
• eof_action: This parameter determines what happens when the overlay stream ends.
  • eof_action=repeat (default): Repeats the last frame of the overlay until the main video ends.
  • eof_action=pass: Stops rendering the overlay and lets the main video continue playing cleanly.
  • eof_action=endall: Terminates the entire output stream as soon as the overlay ends.

To use eof_action instead of cutting the video short, modify the filter like this:

ffmpeg -i main_60fps.mp4 -i overlay_24fps.mp4 -filter_complex \
"[0:v]fps=60[main]; \
 [1:v]fps=60[over]; \
 [main][over]overlay=x=10:y=10:eof_action=pass" \
-c:a copy output.mp4

By explicitly defining a common frame rate using the fps filter prior to overlaying, you eliminate frame-matching guesswork for FFmpeg, resulting in a perfectly synchronized output video.