Stream FFmpeg RGB frames to Python via stdin

This article explains how to configure and execute an FFmpeg command to stream raw RGB video frames directly into a Python script using standard input (stdin). This process allows for high-performance, real-time video frame processing in Python without the overhead of saving temporary image files to your disk.

The FFmpeg Command

To stream raw RGB video frames to stdout (which becomes the stdin of your Python script), use the following FFmpeg command structure:

ffmpeg -i input.mp4 -f rawvideo -pix_fmt rgb24 -

Command Breakdown:

-i input.mp4: Specifies the input video file or stream source.
-f rawvideo: Forces FFmpeg to output raw, demuxed, and uncompressed video packet data.
-pix_fmt rgb24: Sets the pixel format to 24-bit raw RGB (8 bits per channel: Red, Green, Blue). This ensures each pixel is represented by exactly 3 bytes.
-: Instructs FFmpeg to send the output stream to stdout instead of writing it to a file.

Method 1: Piping in the Command Line

If you want to run the FFmpeg command in your terminal and pipe the output directly into a Python script, use the standard pipe operator (|).

The Terminal Command:

ffmpeg -i input.mp4 -f rawvideo -pix_fmt rgb24 - | python processing_script.py

The Python Script (`processing_script.py`):

import sys
import numpy as np

# Set the resolution to match your input video
width = 1920
height = 1080
channels = 3  # 3 channels for RGB
frame_size = width * height * channels

while True:
    # Read the raw frame bytes from stdin
    frame_bytes = sys.stdin.buffer.read(frame_size)
    
    # Break the loop if the stream ends or frame size is incomplete
    if len(frame_bytes) != frame_size:
        break
    
    # Convert raw bytes to a NumPy array for processing
    frame = np.frombuffer(frame_bytes, dtype=np.uint8).reshape((height, width, channels))
    
    # Your image processing code goes here
    # Example: print the RGB value of the top-left pixel
    print(f"Top-left pixel RGB: {frame[0, 0]}")

Method 2: Launching FFmpeg Inside Python (Recommended)

To keep your application self-contained, you can use Python’s subprocess module to spawn the FFmpeg process and read from its stdout pipe directly within the script.

import subprocess
import numpy as np

# Video configuration
input_file = 'input.mp4'
width = 1920
height = 1080
channels = 3
frame_size = width * height * channels

# Build the FFmpeg command
command = [
    'ffmpeg',
    '-i', input_file,
    '-f', 'rawvideo',
    '-pix_fmt', 'rgb24',
    '-'
]

# Start the subprocess
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, bufsize=10**8)

try:
    while True:
        # Read raw bytes directly from the process stdout
        frame_bytes = process.stdout.read(frame_size)
        
        if len(frame_bytes) != frame_size:
            break
        
        # Reshape the byte buffer into an image array
        frame = np.frombuffer(frame_bytes, dtype=np.uint8).reshape((height, width, channels))
        
        # Perform frame processing here
        
finally:
    process.terminate()
    process.wait()