How to Build an Automated Testing Suite with FATE

This guide provides a comprehensive walkthrough for building an automated testing pipeline using the FFmpeg Automated Testing Environment (FATE). FATE is FFmpeg's native regression testing framework, designed to verify media decoding, encoding, demuxing, muxing, and filtering against known reference files. You will learn how to set up the necessary environment, synchronize test media suites, execute and configure existing regression suites, define custom test cases, and integrate FATE into continuous integration (CI) workflows for continuous regression tracking.

Prerequisites and Build Environment

Before running or developing FATE tests, you must compile FFmpeg from source with the relevant dependencies enabled.

Install Build Essentials: Ensure your system has a C compiler (gcc or clang), make, rsync (to download sample media), and git.

Clone the FFmpeg Source:

git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg
cd ffmpeg

Configure the Build: Run ./configure with the specific configurations, external libraries, and debug flags required for your test environment:
```
./configure --enable-gpl --enable-nonfree
make -j$(nproc)
```

Synchronizing the Test Suite Samples

FATE uses a standardized repository of media files to test decoders, demuxers, and filters against reference output.

To download the sample files locally, use the built-in rsync target:

make fate-rsync SAMPLES=fate-suite/

To persist this location across builds, specify the sample directory during configuration:

./configure --samples=fate-suite/

Executing FATE Tests

Once the samples are downloaded, you can trigger test runs using make.

Run the Full Test Suite:
```
make fate
```
Run a Specific Subcategory: Target specific components to save time during localized development:
```
make fate-flac
make fate-h264
make fate-lavf
```
Parallel Execution: Accelerate testing across multiple CPU cores:
```
make fate -j$(nproc)
```

Adding Custom Tests to the FATE Suite

FATE tests are defined in the tests/fate/ directory inside corresponding .mak files (e.g., tests/fate/video.mak, tests/fate/audio.mak).

Place Your Test Sample: Add the media file to your local fate-suite/ directory under an appropriate subfolder (e.g., fate-suite/custom/sample.mov).
Define the Test Case: Open the relevant .mak file in tests/fate/ and define your test rule using standard FATE macros:
```
FATE_CUSTOM += fate-custom-mov
fate-custom-mov: CMD = framecrc -i $(TARGET_SAMPLES)/custom/sample.mov -an

FATE_SAMPLES_AVCONV += $(FATE_CUSTOM)
fate-custom: $(FATE_CUSTOM)
```
- CMD types:
  - framecrc: Outputs per-frame checksums, ideal for video/audio decoding verification.
  - md5 or crc: Generates a single checksum over the entire stream.
  - transcode: Verifies encoding/filtering pipelines.
Generate Reference Files: FATE matches execution outputs against files stored in tests/ref/fate/. Generate the baseline reference file:
```
make fate-custom-mov GEN=1
```
This creates tests/ref/fate/fate-custom-mov, which tracks the expected hashes. Commit this reference file alongside your code.

Integrating FATE into CI/CD Pipelines

To automate execution on every commit, integrate FATE into platforms like GitHub Actions or GitLab CI.

GitHub Actions Workflow Example

Create .github/workflows/fate.yml:

name: FATE Automated Tests

on: [push, pull_request]

jobs:
  run-fate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Source Code
        uses: actions/checkout@v4

      - name: Install Build Dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y yasm nasm rsync build-essential

      - name: Cache FATE Samples
        uses: actions/cache@v4
        with:
          path: fate-suite/
          key: fate-samples-${{ hashFiles('tests/fate/**') }}
          restore-keys: fate-samples-

      - name: Configure FFmpeg
        run: ./configure --samples=fate-suite/

      - name: Build FFmpeg
        run: make -j$(nproc)

      - name: Sync FATE Samples
        run: make fate-rsync SAMPLES=fate-suite/

      - name: Execute Tests
        run: make fate THREADS=$(nproc)

Running Dedicated FATE Client Instances

For distributed, multi-platform testing, FFmpeg includes a client script (tests/fate.sh) designed to run automatically via cron jobs or daemons and report results to a central reporting server:

Copy the configuration template:

cp doc/fate_config.sh.template fate_config.sh

Edit fate_config.sh to define workdir, samples, git_clean, and build flags.
Run the harness:
```
tests/fate.sh /path/to/fate_config.sh
```

This executes a clean checkout, applies updates, runs the entire test matrix, and generates a structured log of failures for automated tracking.