Planar Tracking and Screen Replacement in Blender

Planar tracking is an essential visual effects technique used to follow the motion, perspective, and distortion of flat surfaces within video footage. This guide explains the core concepts of planar tracking and provides a straightforward, step-by-step workflow for replacing a moving phone screen using Blender's built-in Motion Tracking and Compositing tools.

What is Planar Tracking?

Planar tracking involves tracking multiple points on a flat, two-dimensional surface across successive video frames. Unlike standard point tracking, which only follows individual X and Y positions, planar tracking calculates position, rotation, scale, shear, and perspective changes simultaneously. This allows visual effects artists to seamlessly map new images, videos, or user interfaces onto surfaces like phone screens, computer monitors, billboards, and walls.

Step 1: Import and Prepare Footage

  1. Open Blender, switch to the VFX workspace, and select the Movie Clip Editor.
  2. Click Open and load your video clip.
  3. Click Set Scene Frames to match the scene duration with the footage length.
  4. Click Prefetch to load the video frames into memory for smooth playback.

Step 2: Track the Four Corners

  1. Navigate to the first frame where the phone screen is fully visible.
  2. Hold Ctrl and left-click near each of the four corners of the phone screen to add four separate tracking markers.
  3. In the Track panel on the right, set the motion model to Perspective.
  4. Enable Normalize under the Match settings to account for lighting changes.
  5. Select all four markers and press Ctrl + T to track the markers forward through the timeline. Make manual adjustments to any marker that slips off target.

Step 3: Create the Plane Track

  1. With all four tracking markers selected, navigate to the Solve panel in the left sidebar.
  2. Under the Plane Track section, click Create Plane Track.
  3. A planar grid overlay will appear over the tracked points.
  4. Drag the four corner handles of the planar outline so that they line up precisely with the physical edges of the phone screen.

Step 4: Composite the New Screen

  1. Switch to the Compositing workspace and check the Use Nodes box.
  2. Add a Movie Clip node (Shift + A > Input > Movie Clip) and select your background video.
  3. Add an Image node (Shift + A > Input > Image) to load the replacement screen graphic or animation.
  4. Add a Plane Track Deform node (Shift + A > Distort > Plane Track Deform).
  5. In the Plane Track Deform node, select your movie clip, active camera, and the plane track you generated.
  6. Connect the Image output of your replacement screen into the Image input of the Plane Track Deform node.
  7. Add an Alpha Over node (Shift + A > Color > Alpha Over).
  8. Connect the original footage (Movie Clip output) to the top image input of the Alpha Over node, and connect the deformed screen (Plane Track Deform output) to the bottom image input.
  9. Connect the Alpha Over output to the Composite and Viewer nodes to see the final result.

Step 5: Final Adjustments

To make the replacement look natural, match the screen's brightness and contrast using a Color Balance or RGB Curves node between the screen image and the deformer. If fingers occlude the screen during the shot, use Blender's Masking tools in the Movie Clip Editor to create a foreground mask, feeding it into the factor input of an additional Alpha Over node to place the fingers back over the replacement screen.