Rolling Shutter Simulation in GLSL Post-Processing

This article outlines the scanline temporal interpolation algorithm used to simulate CMOS rolling shutter artifacts within a GLSL post-processing pipeline. It explains the physical mechanics of row-by-row sensor readout, details the mathematical formulation required to warp image space based on scanline timing, and provides a direct GLSL implementation utilizing motion vectors or camera transform interpolation.

The Rolling Shutter Mechanism

CMOS image sensors capture images by exposing and reading out scanlines sequentially from the top of the sensor to the bottom, rather than exposing the entire frame simultaneously. When the camera or objects in the scene move rapidly during this readout duration, vertical or horizontal spatial distortion occurs, producing skew, wobble (the "jello" effect), or spatial shearing.

The Scanline Temporal Interpolation Algorithm

To replicate rolling shutter in GLSL post-processing, each horizontal scanline must be sampled at a time offset relative to its vertical position on the screen. The standard approach utilizes Scanline-Dependent Motion Vector Warping or Scanline Camera Matrix Interpolation.

1. Temporal Offset Calculation

First, calculate the normalized readout time offset \(\tau\) for each pixel based on its vertical texture coordinate \(v \in [0, 1]\):

\[\tau = v \cdot T_{\text{readout}}\]

Where \(T_{\text{readout}}\) represents the fraction of the total frame duration required for the sensor to read out from top to bottom (typically between \(0.5\) and \(1.0\)).

2. Vector Displacement

In a deferred or post-processing pipeline that provides a per-pixel motion vector buffer, the screen-space velocity \(\vec{V}\) is scaled by the temporal offset. The distorted sampling coordinate \(\text{UV}_{\text{distorted}}\) is computed as:

\[\text{UV}_{\text{distorted}} = \text{UV} - (\vec{V} \cdot \tau)\]

For scenes dominated by camera rotation or translation where per-pixel motion vectors are not present, the algorithm linearly interpolates (lerp) or spherically interpolates (slerp) between the camera's view-projection matrix at the start of the frame (\(M_{\text{start}}\)) and the end of the frame (\(M_{\text{end}}\)) based on the vertical screen coordinate \(y\):

\[M(y) = \text{mix}(M_{\text{start}}, M_{\text{end}}, y)\]

The pixel then re-projects into world space using the standard depth buffer and reconstructs its position through \(M(y)\).

GLSL Implementation

Below is a GLSL fragment shader implementation using per-pixel screen-space velocity vectors to displace the original frame texture:

#version 330 core

in vec2 vTexCoord;
out vec4 FragColor;

uniform sampler2D uSceneColor;
uniform sampler2D uVelocityTexture; // Screen-space velocity buffer

uniform float uReadoutTime; // Sensor readout duration (0.0 to 1.0)
uniform float uDirection;   // 1.0 for top-to-bottom, -1.0 for bottom-to-top

void main()
{
    // Determine the normalized scanline coordinate along the readout axis
    float scanline = (uDirection > 0.0) ? vTexCoord.y : (1.0 - vTexCoord.y);

    // Calculate temporal weight for the current scanline
    float timeOffset = scanline * uReadoutTime;

    // Fetch the 2D velocity vector for the current pixel
    vec2 velocity = texture(uVelocityTexture, vTexCoord).xy;

    // Displace UV coordinates along the motion direction proportionally to scanline time
    vec2 distortedUV = vTexCoord - (velocity * timeOffset);

    // Prevent sampling outside the texture bounds
    distortedUV = clamp(distortedUV, vec2(0.0), vec2(1.0));

    // Sample the scene using the distorted coordinates
    FragColor = texture(uSceneColor, distortedUV);
}

Enhancing Artifact Accuracy

  • Bidirectional Scanlines: High-speed sensors or specific camera configurations read lines in alternating directions or horizontally. Adjusting the readout vector from vTexCoord.y to vTexCoord.x simulates horizontal rolling shutter commonly found in vertically oriented mobile sensors.
  • Secondary Occlusion Handling: Rapid displacement can cause gaps or overlapping geometries. Sampling both forward and backward motion vectors or applying a spatial blur weighted by length(velocity) helps mask coordinate stretching artifacts that occur during extreme motion.