Convert Pixel Formats with FFmpeg libswscale

Converting pixel formats of a decoded video frame is a common task when working with FFmpeg, especially when preparing raw frames (such as AV_PIX_FMT_YUV420P) for rendering, hardware display, or encoding into formats like AV_PIX_FMT_RGB24 or AV_PIX_FMT_RGBA. This guide demonstrates how to use the libswscale library to initialize a scaling context, allocate a target frame, perform the conversion using sws_scale, and properly manage memory.

1. Include Required Headers

Ensure your project links against libswscale, libavutil, and libavcodec. Include the following headers in your C or C++ source file:

#include <libavutil/frame.h>
#include <libavutil/imgutils.h>
#include <libswscale/swscale.h>

2. Initialize the SwsContext

The conversion is handled by an instance of SwsContext. You initialize this context using sws_getContext(), providing the source dimensions, source pixel format, destination dimensions, destination pixel format, and the interpolation algorithm.

int src_width = src_frame->width;
int src_height = src_frame->height;
enum AVPixelFormat src_pix_fmt = src_frame->format;

int dst_width = src_width; // Change if scaling is needed
int dst_height = src_height;
enum AVPixelFormat dst_pix_fmt = AV_PIX_FMT_RGB24;

struct SwsContext *sws_ctx = sws_getContext(
    src_width,
    src_height,
    src_pix_fmt,
    dst_width,
    dst_height,
    dst_pix_fmt,
    SWS_BILINEAR, // Scaling algorithm (e.g., SWS_BILINEAR, SWS_BICUBIC)
    NULL,
    NULL,
    NULL
);

if (!sws_ctx) {
    // Handle initialization error
}

3. Allocate the Destination Frame

You must allocate an AVFrame and back it with an allocated buffer to hold the converted pixel data.

AVFrame *dst_frame = av_frame_alloc();
if (!dst_frame) {
    // Handle allocation error
}

dst_frame->width = dst_width;
dst_frame->height = dst_height;
dst_frame->format = dst_pix_fmt;

// Allocate memory buffers for the destination frame
int ret = av_frame_get_buffer(dst_frame, 0);
if (ret < 0) {
    // Handle buffer allocation error
}

Alternatively, you can manually allocate a contiguous raw buffer using av_image_alloc() if you only need the raw byte array rather than a fully initialized AVFrame.

4. Execute the Conversion with sws_scale

Once the context and frames are ready, call sws_scale() to perform the conversion. Pass the pointers to the slice data and line sizes from both the source and destination frames.

sws_scale(
    sws_ctx,
    (const uint8_t * const *)src_frame->data,
    src_frame->linesize,
    0,                  // Starting slice row
    src_height,         // Slice height
    dst_frame->data,
    dst_frame->linesize
);

After sws_scale executes, dst_frame->data[0] contains the converted image data.

5. Clean Up Resources

When you are finished with conversion, free the allocated resources to avoid memory leaks. Reusing the SwsContext across frames of the same format and dimensions is recommended for performance.

sws_freeContext(sws_ctx);
av_frame_free(&dst_frame);