How to Write a Custom FATE Test in FFmpeg

This guide explains how to create, configure, and generate reference files for a custom FFmpeg Automated Testing Environment (FATE) test when adding a new audio or video filter. By registering your test target within FFmpeg's Makefiles, declaring appropriate dependencies, generating a deterministic reference output (typically a framecrc), and running the test suite, you ensure your new filter maintains output integrity and prevents future regressions.

1. Identify the Correct Test Makefile

FATE filter tests are located inside the tests/fate/ directory. Depending on the type of filter you created, open the corresponding Makefile:

  • Video filters: tests/fate/filter-video.mak
  • Audio filters: tests/fate/filter-audio.mak

2. Declare Dependencies and Test Targets

FFmpeg uses conditional variables based on build configuration to ensure tests only run when the required filters, demuxers, and protocols are compiled.

Add your test to the appropriate Makefile using the ALLYES macro to guard the test case:

FATE_MYFILTER-$(call ALLYES, MYFILTER_FILTER TESTSRC_FILTER) += fate-filter-myfilter
FATE_FILTER_VIDEO += $(FATE_MYFILTER-yes)

In this example, MYFILTER_FILTER represents your newly implemented filter, and TESTSRC_FILTER is a synthetic video source used as the test input.

3. Define the Command Execution Rule

Next, define the execution command (CMD) for your test. Most filter tests use framecrc to output checksums of the processed frames, which provides a compact, deterministic output without storing large raw media files.

Below the target definition, declare the command:

fate-filter-myfilter: CMD = framecrc -lavfi "testsrc=duration=1:size=320x240:rate=25,myfilter=param1=value1"

If your test relies on existing media samples rather than a generated synthetic source like testsrc, reference the sample file via $(TARGET_SAMPLES):

fate-filter-myfilter: CMD = framecrc -auto_conversion_filters -i $(TARGET_SAMPLES)/path/to/sample.ext -vf "myfilter"

Ensure any external input samples specified are already available in the FATE sample suite.

4. Generate the Reference Output

FATE verifies correctness by comparing the command's standard output against a committed reference file stored in tests/ref/fate/.

Run make with the GEN=1 flag targeting your specific test to create the baseline file:

make fate-filter-myfilter GEN=1

This command generates tests/ref/fate/fate-filter-myfilter. Open this generated file in a text editor to verify that the frame timestamps, channel layouts, pixel formats, and CRC checksums accurately represent the intended behavior of your filter.

5. Validate the Test

Once the reference file exists, run the test without generation flags to confirm that it executes and passes:

make fate-filter-myfilter

If the filter runs and the calculated checksums match the reference file, the terminal will display TEST filter-myfilter followed by a success state. You can also run the broader filter test category to confirm no unintended side-effects occurred:

make fate-filter-video

Commit both the modified Makefile (tests/fate/filter-video.mak) and the newly generated reference file (tests/ref/fate/fate-filter-myfilter) alongside your filter implementation.