How to Add a Custom Container Demuxer to FFmpeg

To support a new, custom container format in FFmpeg, you must write a demuxer (implemented via the AVInputFormat struct). While codec bitstreams use “parsers” to split raw data into packets, container formats are handled by demuxers, which parse the file structure, extract metadata, and separate audio/video streams. This article provides a direct, step-by-step technical guide on how to write, integrate, and compile a custom demuxer inside the FFmpeg source code.

Step 1: Create the Source File

Navigate to the FFmpeg source directory and create a new C file for your demuxer in the libavformat directory:

touch libavformat/mycontainerdec.c

Step 2: Implement the Core Demuxer Functions

Open the newly created file and define the mandatory functions required by FFmpeg to probe, read headers, and extract packets from your format.

#include "avformat.h"
#include "internal.h"
#include "libavutil/intreadwrite.h"

// 1. Probe function: Checks if the input file matches your container format
static int my_container_probe(const AVProbeData *p)
{
    // Analyze magic bytes or signature. Return 100 for a perfect match.
    if (p->buf_size < 4)
        return 0;
    
    if (AV_RL32(p->buf) == MKTAG('M', 'Y', 'C', 'F')) { // "MYCF" Magic Bytes
        return AVPROBE_SCORE_MAX;
    }
    return 0;
}

// 2. Header reading: Parses metadata, defines streams, and initializes decoders
static int my_container_read_header(AVFormatContext *s)
{
    AVStream *st;
    AVIOContext *pb = s->pb;

    // Skip the 4-byte magic number
    avio_skip(pb, 4);

    // Create a new stream (e.g., video stream)
    st = avformat_new_stream(s, NULL);
    if (!st)
        return AVERROR(ENOMEM);

    // Set codec parameters
    st->codecpar->codec_type = AVMEDIA_TYPE_VIDEO;
    st->codecpar->codec_id   = AV_CODEC_ID_H264; // Example codec
    
    // Set timebase (e.g., 90kHz clock)
    avpriv_set_pts_info(st, 64, 1, 90000);

    return 0;
}

// 3. Packet reading: Reads raw packet data from the stream and assigns timestamps
static int my_container_read_packet(AVFormatContext *s, AVPacket *pkt)
{
    AVIOContext *pb = s->pb;
    int ret, packet_size;
    int64_t pts;

    if (avio_feof(pb))
        return AVERROR_EOF;

    // Custom logic to read packet size and timestamp from container header
    packet_size = avio_rl32(pb); 
    pts         = avio_rl64(pb);

    // Allocate and populate the packet
    ret = av_get_packet(pb, pkt, packet_size);
    if (ret < 0)
        return ret;

    pkt->pts = pts;
    pkt->stream_index = 0; // Associates packet with our stream created in read_header

    return 0;
}

// 4. Seek function (Optional but recommended)
static int my_container_read_seek(AVFormatContext *s, int stream_index, 
                                  int64_t timestamp, int flags)
{
    // Implement seeking logic using avio_seek() here
    return 0; 
}

Step 3: Define the AVInputFormat Structure

At the bottom of your mycontainerdec.c file, declare the AVInputFormat structure that registers your custom functions to the FFmpeg engine:

const AVInputFormat ff_mycontainer_demuxer = {
    .name           = "mycontainer",
    .long_name      = NULL_IF_CONFIG_SMALL("My Custom Container Format"),
    .read_probe     = my_container_probe,
    .read_header    = my_container_read_header,
    .read_packet    = my_container_read_packet,
    .read_seek      = my_container_read_seek,
    .flags          = AVFMT_GENERIC_INDEX,
};

Step 4: Register the Demuxer in the Build System

To compile your demuxer, you must register it inside FFmpeg’s build configuration files.

Modify libavformat/Makefile: Open libavformat/Makefile, find the list of demuxers, and add your file alphabetically:
```
OBJS-$(CONFIG_MYCONTAINER_DEMUXER)      += mycontainerdec.o
```
Modify libavformat/allformats.c: Depending on your FFmpeg version, demuxers are auto-registered or manually listed. In modern versions, compilation is automatically handled by the build script once declared in the Makefile and matching the name template ff_mycontainer_demuxer.

Step 5: Configure and Compile

Run the configure script to detect your new format, then compile FFmpeg:

./configure --enable-demuxer=mycontainer
make -j$(nproc)

Step 6: Test Your Demuxer

Verify that FFmpeg recognizes your new format by running the following command to check if it appears in the supported formats list:

./ffmpeg -formats | grep mycontainer

You can now parse your custom container file using:

./ffmpeg -f mycontainer -i input.mycf output.mp4