GLSL Pixel Reconstruction Along Motion Vectors

This article provides a technical overview of how pixel reconstruction filtering is implemented along motion vector trajectories in OpenGL Shading Language (GLSL). It examines the end-to-end fragment shading pipeline commonly used in Temporal Anti-Aliasing (TAA) and temporal upscaling, including reading screen-space velocity, sampling historical buffers along the motion path, resolving historical color data with cubic filtering, applying neighborhood color bounding to prevent ghosting, and blending the temporal accumulation.

1. Motion Vector Retrieval and Reprojection

Pixel reconstruction begins by determining where the current fragment was located in the preceding frame. This is achieved using a screen-space velocity buffer generated during the geometry pass or derived from depth:

// Fetch 2D motion vector (stored in UV space or mapped from screen space)
vec2 velocity = texture(uVelocityTexture, vUV).xy;

// Calculate the previous frame UV coordinates
vec2 historyUV = vUV - velocity;

If the motion vector buffer stores delta coordinates in normalized device coordinates (NDC), the vector must be mapped to the \([0, 1]\) texture coordinate domain before subtracting.

2. High-Order Historical Reconstruction

Standard bilinear interpolation degrades image sharpness across multiple frames. To counteract blur, reconstruction filtering along the trajectory samples the historical texture using higher-order filtering kernels, such as a Catmull-Rom spline or Mitchell-Netravali filter, using a 5-tap or 9-tap optimized gather:

vec4 SampleTextureCatmullRom(sampler2D tex, vec2 uv, vec2 texSize) {
    vec2 samplePos = uv * texSize;
    vec2 tc = floor(samplePos - 0.5) + 0.5;
    vec2 f = samplePos - tc;

    vec2 w0 = f * (-0.5 + f * (1.0 - 0.5 * f));
    vec2 w1 = 1.0 + f * f * (-2.5 + 1.5 * f);
    vec2 w2 = f * (0.5 + f * (2.0 - 1.5 * f));
    vec2 w3 = f * f * (-0.5 + 0.5 * f);

    vec2 w12 = w1 + w2;
    vec2 tc12 = tc + w2 / w12;

    vec2 tc0 = tc - 1.0;
    vec2 tc3 = tc + 2.0;

    vec4 color = vec4(0.0);
    color += texture(tex, vec2(tc12.x, tc0.y) / texSize) * (w12.x * w0.y);
    color += texture(tex, vec2(tc0.x, tc12.y) / texSize) * (w0.x * w12.y);
    color += texture(tex, vec2(tc12.x, tc12.y) / texSize) * (w12.x * w12.y);
    color += texture(tex, vec2(tc3.x, tc12.y) / texSize) * (w3.x * w12.y);
    color += texture(tex, vec2(tc12.x, tc3.y) / texSize) * (w12.x * w3.y);

    return color / (w12.x * w0.y + w0.x * w12.y + w12.x * w12.y + w3.x * w12.y + w12.x * w3.y);
}

3. Trajectory-Aligned Accumulation (Motion Blur)

When performing temporal reconstruction for motion blur rather than anti-aliasing, multiple samples are integrated explicitly along the direction of the motion vector within the current frame:

int numSamples = 8;
vec4 motionBlurredColor = vec4(0.0);
float totalWeight = 0.0;

for (int i = 0; i < numSamples; ++i) {
    float t = (float(i) / float(numSamples - 1)) - 0.5;
    vec2 sampleUV = vUV + velocity * t;
    
    // Gaussian or tent weighting along the trajectory
    float weight = exp(-2.0 * t * t);
    motionBlurredColor += texture(uCurrentColorTexture, sampleUV) * weight;
    totalWeight += weight;
}

motionBlurredColor /= totalWeight;

4. Neighborhood Clamping and Variance Clipping

To mitigate ghosting artifacts caused by occlusion boundaries, the reconstructed history sample is bounded using the local neighborhood of the current frame. A \(3 \times 3\) footprint is sampled around the current pixel:

vec3 m1 = vec3(0.0);
vec3 m2 = vec3(0.0);

for (int x = -1; x <= 1; ++x) {
    for (int y = -1; y <= 1; ++y) {
        vec3 c = textureOffset(uCurrentColorTexture, vUV, ivec2(x, y)).rgb;
        m1 += c;
        m2 += c * c;
    }
}

// Calculate mean and variance
vec3 mean = m1 / 9.0;
vec3 stdDev = sqrt(max(vec3(0.0), (m2 / 9.0) - (mean * mean)));

// Define axis-aligned bounding box in color space
float gamma = 1.0; // Variance scale
vec3 bMin = mean - gamma * stdDev;
vec3 bMax = mean + gamma * stdDev;

// Clip the historical sample to the bounding box
vec3 clippedHistory = clamp(historyColor.rgb, bMin, bMax);

5. Final Temporal Blend

The final step blends the current pixel data with the clamped historical data. The blend factor \(\alpha\) is modulated dynamically based on whether the history coordinates are valid (within the screen bounds) and the velocity magnitude:

void main() {
    vec2 velocity = texture(uVelocityTexture, vUV).xy;
    vec2 historyUV = vUV - velocity;

    vec4 currentColor = texture(uCurrentColorTexture, vUV);
    vec4 historyColor = SampleTextureCatmullRom(uHistoryTexture, historyUV, uTexSize);

    // Apply color space constraints (e.g., in YCoCg or RGB)
    vec3 clampedHistory = clamp(historyColor.rgb, bMin, bMax);

    // Base temporal blend weight (typically 0.05 to 0.1 for the current frame)
    float blendWeight = 0.1;

    // Discard history if reprojection falls outside screen margins
    if (any(lessThan(historyUV, vec2(0.0))) || any(greaterThan(historyUV, vec2(1.0)))) {
        blendWeight = 1.0;
    }

    fragColor = vec4(mix(clampedHistory, currentColor.rgb, blendWeight), 1.0);
}

This pipeline produces a temporally stable result by balancing the high-frequency spatial detail of the current frame with the anti-aliased, jitter-accumulated data of prior frames along their true geometric motion vectors.