Extract Filtered Frames with FFmpeg Buffer Sink API

This article explains how to retrieve filtered video or audio frames from an FFmpeg filtergraph using the buffersink or abuffersink media sinks in C. We will cover the essential initialization steps, the frame extraction loop using av_buffersink_get_frame, and resource management to ensure a leak-free implementation.

Understanding the Buffer Sink

In an FFmpeg filtergraph, data flows from one or more sources (buffersrc or abuffersrc) through a chain of filters, and ends at one or more sinks (buffersink for video or abuffersink for audio). The sink acts as the end-point of the graph where the fully processed frames are temporarily stored, waiting to be read by your application.

Step-by-Step Implementation

To successfully extract frames from a filtergraph, you must feed frames into the source, trigger the processing, and pull the results from the sink.

1. Locate the Sink Filter Context

When creating and configuring your filtergraph, you must keep a reference to the sink’s AVFilterContext. This context pointer is required to pull the processed frames.

AVFilterContext *buffersink_ctx; // Retrieved during filtergraph configuration

2. The Frame Extraction Loop

Once you write a raw frame to the filtergraph using av_buffersrc_add_frame (or av_buffersrc_write_frame), the filtergraph processes the data. You must then retrieve the output frame(s) using av_buffersink_get_frame in a loop. A single input frame can produce zero, one, or multiple output frames depending on the filters used (e.g., fps filters, deinterlacing, or decimation).

Here is the standard implementation loop:

AVFrame *filt_frame = av_frame_alloc();
if (!filt_frame) {
    // Handle memory allocation failure
}

int ret;
while (1) {
    // Pull a filtered frame from the sink
    ret = av_buffersink_get_frame(buffersink_ctx, filt_frame);
    
    if (ret == AVERROR(EAGAIN)) {
        // EAGAIN: More input frames are required before the sink can output a frame
        break;
    }
    if (ret == AVERROR_EOF) {
        // EOF: The end of the stream has been reached; no more frames will be produced
        break;
    }
    if (ret < 0) {
        // An actual error occurred
        fprintf(stderr, "Error pulling frame from filtergraph\n");
        break;
    }

    // Process your filtered frame here (e.g., play, encode, or save)
    // filt_frame->data contains the processed image/audio data

    // You must unreference the frame to free its buffers before the next loop iteration
    av_frame_unref(filt_frame);
}

// Clean up the frame allocation
av_frame_free(&filt_frame);

3. Handling Audio vs. Video Sinks

While the extraction loop code is identical for both audio and video, you must configure the sink properties differently during filtergraph setup:

Key Considerations