Side-by-Side Video and Audio Visualizer in FFmpeg

This article explains how to take an existing media file containing both audio and video, generate an animated audio visualization from the sound track, and place the original video and the visualizer side-by-side using FFmpeg. By leveraging FFmpeg's filter_complex engine with filters such as showwaves, showfreqs, scale, and hstack, you can generate clean, synchronized side-by-side presentations without needing complex video editing software.

The Basic Command

To place the video on the left and a waveform visualizer on the right, ensure that both visual elements have the exact same height so they can be horizontally stacked using the hstack filter.

ffmpeg -i input.mp4 -filter_complex \
"[0:v]scale=960:1080[vid]; \
 [0:a]showwaves=s=960x1080:mode=line:colors=cyan[viz]; \
 [vid][viz]hstack=inputs=2[v]" \
-map "[v]" -map 0:a -c:v libx264 -c:a copy output.mp4

Breakdown of the Command

  • -i input.mp4: Defines your input file containing both video and audio streams.
  • [0:v]scale=960:1080[vid]: Takes the original video stream (0:v) and resizes it to a width of 960 pixels and a height of 1080 pixels, tagging the output as [vid].
  • [0:a]showwaves=s=960x1080:mode=line:colors=cyan[viz]: Converts the audio stream (0:a) into an animated waveform visualizer with dimensions matching the height of the scaled video (960x1080), outputting it as [viz].
  • [vid][viz]hstack=inputs=2[v]: Stacks the scaled video ([vid]) and the visualization ([viz]) side-by-side into a single 1920x1080 output stream labeled [v].
  • -map "[v]" -map 0:a: Directs FFmpeg to use the newly combined video stream [v] and include the original, unaltered audio track 0:a.
  • -c:v libx264 -c:a copy: Encodes the final video stream using the H.264 codec while copying the original audio stream directly to avoid quality loss.

Using a Frequency Spectrum Visualizer

If you prefer a bar or frequency spectrum display instead of standard waveforms, substitute showwaves with showfreqs:

ffmpeg -i input.mp4 -filter_complex \
"[0:v]scale=960:1080[vid]; \
 [0:a]showfreqs=s=960x1080:mode=bar:ascale=log:fscale=log:colors=yellow|red[viz]; \
 [vid][viz]hstack=inputs=2[v]" \
-map "[v]" -map 0:a -c:v libx264 -c:a copy output.mp4

Keeping Original Video Proportions

If you do not want to force-stretch your original video, you can preserve its aspect ratio while scaling by using -1 for one dimension, then pad or scale the visualizer accordingly:

ffmpeg -i input.mp4 -filter_complex \
"[0:v]scale=-1:720[vid]; \
 [0:a]showwaves=s=640x720:mode=cline:colors=white[viz]; \
 [vid][viz]hstack=inputs=2[v]" \
-map "[v]" -map 0:a -c:v libx264 -c:a copy output.mp4

In this case, the scale=-1:720 filter automatically sets the width to maintain the original aspect ratio while fixing the height to 720 pixels, matching the 720-pixel height of the visualizer.