Use MediaCodec Hardware Decoder with FFmpeg on Android

Integrating Android’s native MediaCodec hardware decoder into an FFmpeg workflow allows developers to leverage hardware-accelerated video decoding for improved performance, lower latency, and reduced CPU and battery consumption. This article provides a direct, technical guide on how to compile FFmpeg with MediaCodec support, initialize the Java Virtual Machine (JVM) within your C/C++ codebase, select the appropriate hardware decoder, and retrieve decoded frames efficiently.

1. Compile FFmpeg with MediaCodec Support

To use MediaCodec within FFmpeg, you must compile FFmpeg with the appropriate JNI (Java Native Interface) and MediaCodec flags enabled. When configuring your FFmpeg build script for Android, ensure you include the following flags:

--enable-jni \
--enable-mediacodec \
--enable-decoder=h264_mediacodec \
--enable-decoder=hevc_mediacodec \
--enable-decoder=vp9_mediacodec

These flags instruct FFmpeg to compile the JNI helper utilities and map Android’s system MediaCodec APIs to FFmpeg’s decoder interface under the *_mediacodec names.

2. Initialize the JVM in Your Native Code

Because MediaCodec is a Java-based API in Android, FFmpeg’s native C/C++ code needs access to the Java Virtual Machine (JVM) to invoke it. You must register the JVM pointer with FFmpeg before performing any decoding operations.

In your JNI initialization function (usually JNI_OnLoad), pass the JavaVM pointer to FFmpeg:

#include <libavcodec/jni.h>

jint JNI_OnLoad(JavaVM *vm, void *reserved) {
    // Provide the JVM pointer to FFmpeg
    av_jni_set_java_vm(vm, NULL);
    return JNI_VERSION_1_6;
}

3. Locate and Open the MediaCodec Decoder

Instead of looking up standard software decoders (like h264 or hevc), you must explicitly search for the MediaCodec variant using avcodec_find_decoder_by_name.

#include <libavcodec/avcodec.h>

// Find the H.264 MediaCodec hardware decoder
const AVCodec *codec = avcodec_find_decoder_by_name("h264_mediacodec");
if (!codec) {
    // Fallback to software decoder if hardware decoder is unavailable
    codec = avcodec_find_decoder(AV_CODEC_ID_H264);
}

AVCodecContext *codec_ctx = avcodec_alloc_context3(codec);
if (!codec_ctx) {
    // Handle allocation failure
}

// Open the codec context
if (avcodec_open2(codec_ctx, codec, NULL) < 0) {
    // Handle initialization failure
}

4. Feed and Retrieve Frames in the Loop

Once the decoder is initialized, the standard FFmpeg packet-to-frame loop handles the decoding. The hardware acceleration happens transparently behind the scenes.

AVPacket *packet = av_packet_alloc();
AVFrame *frame = av_frame_alloc();

// Read packets from your AVFormatContext
while (av_read_frame(format_ctx, packet) >= 0) {
    if (packet->stream_index == video_stream_idx) {
        // Send the compressed packet to the hardware decoder
        int response = avcodec_send_packet(codec_ctx, packet);
        if (response < 0) break;

        while (response >= 0) {
            // Retrieve decoded raw frame from the hardware decoder
            response = avcodec_receive_frame(codec_ctx, frame);
            if (response == AVERROR(EAGAIN) || response == AVERROR_EOF) {
                break;
            } else if (response < 0) {
                // Handle error
                break;
            }

            // Frame is now ready. 
            // Note: By default, the frame data is copied back to CPU memory (YUV420P format).
            process_decoded_frame(frame);
        }
    }
    av_packet_unref(packet);
}

av_frame_free(&frame);
av_packet_free(&packet);

5. Advanced: Direct Surface Rendering (Zero-Copy)

By default, FFmpeg copies decoded frame data from the GPU back into CPU memory (YUV buffers). While simple to handle, this transfer limits performance. For maximum efficiency, you can render directly to an Android Surface (e.g., a SurfaceView or TextureView) without copying the pixels back to CPU memory.

To enable direct rendering:

1. Retrieve the ANativeWindow associated with your Android Surface in C/C++.
2. Initialize the FFmpeg decoder context’s hw_device_ctx using AVHWDeviceType set to AV_HWDEVICE_TYPE_MEDIACODEC.
3. Pass the surface handle to the codec context before calling avcodec_open2 using the AVMediaCodecContext structure:

#include <libavcodec/mediacodec.h>

AVMediaCodecContext *media_codec_ctx = av_mediacodec_alloc_context();
// surface is a jobject pointing to your Android Surface
media_codec_ctx->surface = surface; 

codec_ctx->hwaccel_context = media_codec_ctx;

When using direct Surface rendering, calling avcodec_receive_frame will not output raw pixel data. Instead, it triggers the hardware decoder to render the decoded frame directly to the attached screen surface.