How to Write FFmpeg FATE Tests for Custom Filters

This article provides a step-by-step guide on how to write a custom FATE (FFmpeg Automated Testing Environment) test case for a newly developed video or audio filter in FFmpeg. You will learn how to integrate your test into the FFmpeg build system, define the test parameters, generate the expected reference output, and run the automated test suite to validate your filter’s performance and regression safety.

Step 1: Locate the Filter Test Configuration Files

FFmpeg organizes its FATE tests inside the tests/fate/ directory. Depending on the type of filter you have developed, you will need to append your test configuration to the appropriate makefile:

Open the appropriate file in your text editor to prepare for adding your custom test case.

Step 2: Define the Test Case in the Makefile

To add your test, you must declare it using the FFmpeg makefile syntax. This ensures the test only runs if your filter is enabled during the ./configure build process.

Add a block similar to the following at the end of the file (using a hypothetical video filter named myfilter):

FATE_FILTER_VSYNTH-$(call FILTER, MYFILTER) += fate-filter-myfilter
fate-filter-myfilter: CMD = framecrc -i $(SRC) -vf myfilter=param1=value1

In this definition: * FATE_FILTER_VSYNTH-$(call FILTER, MYFILTER) ensures the test is only compiled and run if myfilter is enabled. * fate-filter-myfilter is the unique name of your test case. * CMD defines the test command. framecrc is commonly used for video filters to output a checksum of each decoded frame, which is highly effective for detecting visual regressions. * $(SRC) represents the input test synthetic video sequence automatically managed by FATE.

Step 3: Generate the Reference File

FATE determines if a test passes by comparing the output of your command with an established reference file. You must generate this reference file for your new filter.

Run the following make command from your FFmpeg build directory to automatically generate the reference file:

make fate-filter-myfilter GEN=1

This command runs your filter against the test input and writes the resulting frame checksums to a new reference file located at:

tests/ref/fate/filter-myfilter

Open this newly generated reference file and verify that the format looks correct (typically a list of frame indices, timestamps, and CRC hashes). Commit this reference file to your Git repository along with your code changes.

Step 4: Run and Validate Your Test

Once the reference file is generated, you should verify that the test executes and passes under normal conditions. Run the specific test case using the following command:

make fate-filter-myfilter

If the test is successful, the console will output nothing or a simple success message. If the output deviates from your reference file, the build system will display a diff showing the mismatch, indicating a bug in the filter or an outdated reference file.