Stream Raw RGB Frames from FFmpeg to Python via Stdin

Streaming raw RGB video frames from FFmpeg directly into a Python script allows you to process high-throughput video data in real time without creating temporary files. This guide details how to configure the FFmpeg command line to decode video into an uncompressed rgb24 pixel format and pipe the output to Python using either standard input redirection or the subprocess module.

The FFmpeg Command

To output raw pixel data to standard output (stdout), use the following FFmpeg command:

ffmpeg -i input.mp4 -f rawvideo -pix_fmt rgb24 -

Key arguments explained:

• -i input.mp4: Specifies the input video file (or stream URL/device).
• -f rawvideo: Forces the output format to be a headerless, uncompressed raw video stream.
• -pix_fmt rgb24: Formats the pixel data sequentially as 8-bit Red, Green, and Blue bytes (3 bytes per pixel).
• -: Directs the output to stdout rather than saving it to a file.

Reading the Stream in Python

Because raw video does not include metadata (such as width, height, or framerate) in the stream itself, your Python script must know the video's dimensions beforehand to correctly reconstruct each frame. Each frame consists of exactly width * height * 3 bytes.

Method 1: Launching FFmpeg as a Subprocess (Recommended)

Managing FFmpeg directly inside Python via subprocess.Popen is the cleanest approach. It allows Python to read the video stream via stdout while automatically managing process lifecycles.

import subprocess
import numpy as np

width = 1920
height = 1080
frame_size = width * height * 3

cmd = [
    'ffmpeg',
    '-i', 'input.mp4',
    '-f', 'rawvideo',
    '-pix_fmt', 'rgb24',
    '-'
]

# Start FFmpeg process
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, bufsize=10**8)

while True:
    # Read the exact number of bytes for one frame
    raw_frame = process.stdout.read(frame_size)
    
    if len(raw_frame) != frame_size:
        break  # End of stream or incomplete frame
    
    # Convert the raw bytes to a NumPy array
    frame = np.frombuffer(raw_frame, dtype=np.uint8).reshape((height, width, 3))
    
    # Process the frame (frame contains RGB data)

process.stdout.close()
process.wait()

Method 2: Piping via the Command Line

If you prefer to pipe directly from your terminal using ffmpeg -i input.mp4 -f rawvideo -pix_fmt rgb24 - | python script.py, read from sys.stdin.buffer instead of sys.stdin to handle raw binary data.

import sys
import numpy as np

width = 1920
height = 1080
frame_size = width * height * 3

while True:
    raw_frame = sys.stdin.buffer.read(frame_size)
    
    if len(raw_frame) != frame_size:
        break
        
    frame = np.frombuffer(raw_frame, dtype=np.uint8).reshape((height, width, 3))
    
    # Process frame here