Variance Clipping in GLSL TAA Explained

Variance clipping is a critical stabilization technique used in Temporal Anti-Aliasing (TAA) to eliminate ghosting artifacts caused by disocclusions and rapid lighting changes. Instead of relying on a rigid min/max bounding box that can permit extreme outliers into the temporal blend, variance clipping statistically calculates the mean and standard deviation of color samples within a 3x3 pixel neighborhood in the current frame. By projecting the reprojected history color toward the neighborhood average and clamping it to an axis-aligned bounding box (AABB) derived from standard deviation, TAA preserves sharp edges while strictly rejecting invalid history samples.

The 3x3 Color Neighborhood Sampling

In a standard TAA resolve pass, the current pixel is evaluated alongside its eight immediate neighbors. Because RGB space is non-perceptual and can cause hue shifts during clipping, these samples are typically converted into a decorrelated color space with luminance and chrominance separated, such as YCoCg:

vec3 RGBToYCoCg(vec3 c) {
    return vec3(
         c.r * 0.25 + c.g * 0.5 + c.b * 0.25,
         c.r * 0.5  - c.b * 0.5,
        -c.r * 0.25 + c.g * 0.5 - c.b * 0.25
    );
}

The shader samples the current frame buffer across the \(3\times3\) grid:

vec3 m1 = vec3(0.0); // First moment (sum of colors)
vec3 m2 = vec3(0.0); // Second moment (sum of colors squared)

for (int x = -1; x <= 1; ++x) {
    for (int y = -1; y <= 1; ++y) {
        vec3 color = RGBToYCoCg(texelFetch(currentFrameTex, currentCoord + ivec2(x, y), 0).rgb);
        m1 += color;
        m2 += color * color;
    }
}

Computing Statistical Moments

Rather than tracking strictly the minimum and maximum component values—which can produce an oversized bounding box if high-contrast edges or single-pixel specular highlights are present—variance clipping computes the statistical distribution:

  1. Mean (\(\mu\)): The average color across the 9 samples.
  2. Variance (\(\sigma^2\)): The spread of values around the mean.
  3. Standard Deviation (\(\sigma\)): The square root of variance.

In GLSL, this is expressed as:

const float N = 9.0;
vec3 mu = m1 / N;
vec3 sigma = sqrt(max(vec3(0.0), (m2 / N) - (mu * mu)));

Constructing the Bounding Box

The variance-based AABB is defined around the mean \(\mu\), expanded by a tuning scalar \(\gamma\) (gamma):

\[\text{Box}_{\min} = \mu - \gamma \cdot \sigma\] \[\text{Box}_{\max} = \mu + \gamma \cdot \sigma\]

A \(\gamma\) factor between \(1.0\) and \(1.5\) is typical. A smaller \(\gamma\) aggressively suppresses ghosting at the cost of slight temporal instability or softness, while a larger \(\gamma\) retains more historical temporal samples for smoother edges.

float gamma = 1.0;
vec3 boxMin = mu - gamma * sigma;
vec3 boxMax = mu + gamma * sigma;

Clipping the History Sample

Once the bounding box is established, the reprojected historical color (historyColor) is evaluated against it. While simple component-wise clamping (clamp(historyColor, boxMin, boxMax)) can be used, it often causes color distortion because it alters the vector's direction toward the clipping boundary.

Proper variance clipping finds the intersection of the line segment between the historical sample and the neighborhood center (\(\mu\), or alternatively the current frame's center pixel) with the AABB. This preserves the trajectory of the historical color:

vec3 ClipToAABB(vec3 history, vec3 center, vec3 boxMin, vec3 boxMax) {
    vec3 p_clip = 0.5 * (boxMax + boxMin);
    vec3 e_clip = 0.5 * (boxMax - boxMin) + 0.000001; // Avoid division by zero

    vec3 v_clip = history - p_clip;
    vec3 v_unit = v_clip / e_clip;
    vec3 a_unit = abs(v_unit);
    float ma_unit = max(a_unit.x, max(a_unit.y, a_unit.z));

    if (ma_unit > 1.0) {
        return p_clip + v_clip / ma_unit;
    }
    return history;
}

The resulting clipped color is converted back to RGB and blended with the current frame using an exponential moving average (commonly 90–95% history and 5–10% current frame), delivering a stable, anti-aliased image free of ghosting trails.