Apply Video Filters to WebRTC Streams with Canvas API

This article explains how developers can apply custom real-time video filters and effects to a WebRTC stream using the HTML5 Canvas API. By capturing a camera feed, drawing its frames onto a canvas, manipulating the raw pixel data, and capturing the canvas as a new stream, you can easily transmit modified video over a WebRTC peer connection.

Step 1: Capture the Source Video Stream

First, access the user’s webcam using the getUserMedia API. This stream is attached to an HTML5 <video> element, which plays the video silently in the background to serve as the data source for our canvas.

const video = document.createElement('video');
video.autoplay = true;
video.playsInline = true;
video.muted = true;

navigator.mediaDevices.getUserMedia({ video: true, audio: false })
  .then((stream) => {
    video.srcObject = stream;
  })
  .catch((err) => console.error("Error accessing webcam:", err));

Step 2: Set Up the Canvas and Process Frames

Next, create a <canvas> element and a rendering context. Use a loop powered by requestAnimationFrame to continuously draw the current video frame onto the canvas.

Once the frame is drawn, retrieve its pixel data using getImageData(), loop through the RGBA values to apply your custom effect, and write the modified pixels back using putImageData().

const canvas = document.createElement('canvas');
const ctx = canvas.getContext('2d');

function processVideoFrame() {
  if (video.readyState === video.HAVE_ENOUGH_DATA) {
    // Match canvas dimensions to the video resolution
    if (canvas.width !== video.videoWidth) {
      canvas.width = video.videoWidth;
      canvas.height = video.videoHeight;
    }

    // Draw current video frame to canvas
    ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

    // Get pixel data
    const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);
    const data = imageData.data;

    // Apply a custom filter (e.g., Grayscale)
    for (let i = 0; i < data.length; i += 4) {
      const red = data[i];
      const green = data[i + 1];
      const blue = data[i + 2];
      
      // Calculate luminance
      const grayscale = 0.2126 * red + 0.7152 * green + 0.0722 * blue;

      data[i] = grayscale;     // Red
      data[i + 1] = grayscale; // Green
      data[i + 2] = grayscale; // Blue
    }

    // Write modified pixels back to canvas
    ctx.putImageData(imageData, 0, 0);
  }
  
  // Loop on next animation frame
  requestAnimationFrame(processVideoFrame);
}

// Start processing when video starts playing
video.addEventListener('play', () => {
  requestAnimationFrame(processVideoFrame);
});

Step 3: Capture the Canvas Stream and Send via WebRTC

To send the modified video through WebRTC, use the captureStream() method on the canvas element. This generates a new MediaStream containing the canvas’s visual output at a targeted frame rate. You can then add the track from this stream to your WebRTC RTCPeerConnection.

// Capture the canvas stream at 30 frames per second
const filteredStream = canvas.captureStream(30);
const filteredVideoTrack = filteredStream.getVideoTracks()[0];

// Add the filtered track to your peer connection
const peerConnection = new RTCPeerConnection(configuration);
peerConnection.addTrack(filteredVideoTrack, filteredStream);

Key Optimization Tips