How to Add a Custom Container Demuxer to FFmpeg

This guide explains how to write and integrate a custom container format parser—technically known as a demuxer—into FFmpeg's libavformat library. It covers the core architecture of FFmpeg demuxers, implementing the required detection and parsing callbacks, managing stream allocation and packet extraction, and registering the new format within FFmpeg's build configuration.

1. Understand Demuxing in FFmpeg

In FFmpeg terminology, handling a container format requires an AVInputFormat (demuxer), not an AVCodecParser (which is used for raw elementary stream framing like H.264 Annex B). A demuxer is responsible for reading the container header, identifying media tracks, wrapping payload data into AVPacket instances, and handling stream positioning.

2. Set Up the Source File

Create a new C file inside the libavformat/ directory of the FFmpeg source tree, typically named after your format (e.g., libavformat/customdec.c). Include the required internal headers:

#include "libavutil/intreadwrite.h"
#include "avformat.h"
#include "internal.h"
#include "avio_internal.h"

3. Implement the Probe Function

The probe function determines if an arbitrary input matches your container format. FFmpeg supplies an AVProbeData buffer containing the first few kilobytes of the file. Return a score from 0 to AVPROBE_SCORE_MAX based on confidence:

static int custom_probe(const AVProbeData *p)
{
    // Check for a 4-byte magic signature, e.g., 'C', 'U', 'S', 'T'
    if (AV_RL32(p->buf) == MKTAG('C', 'U', 'S', 'T'))
        return AVPROBE_SCORE_MAX;

    return 0;
}

4. Implement the Header Reader

The read_header function initializes streams, reads track parameters, and sets timestamps. Use AVFormatContext and the AVIOContext (s->pb) to read bytes sequentially.

static int custom_read_header(AVFormatContext *s)
{
    AVIOContext *pb = s->pb;
    AVStream *st;

    // Skip magic bytes if probed earlier
    avio_skip(pb, 4);

    // Read track metadata from container header
    uint32_t num_streams = avio_rl32(pb);

    for (uint32_t i = 0; i < num_streams; i++) {
        st = avformat_new_stream(s, NULL);
        if (!st)
            return AVERROR(ENOMEM);

        st->id = i;
        
        // Example: container specifies codec type (1 = Video, 2 = Audio)
        uint8_t type = avio_r8(pb);
        if (type == 1) {
            st->codecpar->codec_type = AVMEDIA_TYPE_VIDEO;
            st->codecpar->codec_id   = AV_CODEC_ID_H264;
            st->codecpar->width      = avio_rl16(pb);
            st->codecpar->height     = avio_rl16(pb);
        } else if (type == 2) {
            st->codecpar->codec_type  = AVMEDIA_TYPE_AUDIO;
            st->codecpar->codec_id    = AV_CODEC_ID_AAC;
            st->codecpar->sample_rate = avio_rl32(pb);
            st->codecpar->ch_layout.nb_channels = avio_r8(pb);
        }

        // Set time base for the stream (e.g., 1/1000 for milliseconds)
        avpriv_set_pts_info(st, 64, 1, 1000);
    }

    return 0;
}

5. Implement the Packet Reader

The read_packet function is called repeatedly by the playback or transcoding pipeline to fetch individual frames. Read the frame size and metadata, allocate the packet, populate timestamps, and copy payload data:

static int custom_read_packet(AVFormatContext *s, AVPacket *pkt)
{
    AVIOContext *pb = s->pb;

    if (avio_feof(pb))
        return AVERROR_EOF;

    // Read custom frame header
    int stream_idx    = avio_r8(pb);
    int64_t pts       = avio_rl64(pb);
    uint32_t payload_size = avio_rl32(pb);

    // Read payload directly into the packet buffer
    int ret = av_get_packet(pb, pkt, payload_size);
    if (ret < 0)
        return ret;

    pkt->stream_index = stream_idx;
    pkt->pts          = pts;
    pkt->dts          = pts;

    return 0;
}

6. Declare the `AVInputFormat` Structure

Instantiate the AVInputFormat descriptor connecting your callback implementations:

const AVInputFormat ff_custom_demuxer = {
    .name           = "custom",
    .long_name      = NULL_IF_CONFIG_SMALL("Custom Container Format"),
    .read_probe     = custom_probe,
    .read_header    = custom_read_header,
    .read_packet    = custom_read_packet,
    .extensions     = "cst",
    .flags          = AVFMT_GENERIC_INDEX,
};

7. Register the Demuxer in the Build System

To expose the demuxer to the build pipeline, update the following build files:

libavformat/allformats.c: Add an external declaration:
```
extern const AVInputFormat ff_custom_demuxer;
```
libavformat/Makefile: Register the object file under the demuxer configuration flag:
```
OBJS-$(CONFIG_CUSTOM_DEMUXER) += customdec.o
```

8. Compile and Verify

Reconfigure and build FFmpeg:

./configure --enable-demuxer=custom
make -j$(nproc)

Verify that the parser works by inspecting a sample file:

./ffmpeg -v trace -i test.cst