Denoising Ambient Occlusion with Cross-Bilateral GLSL

This article explores the cross-bilateral filtering algorithm used to denoise screen-space ambient occlusion (SSAO) in real-time graphics pipelines. It details how the joint bilateral filter utilizes auxiliary depth and normal buffers to smooth out Monte Carlo or low-sample noise while preserving sharp geometric edges. You will learn the mathematical mechanics of depth-aware edge stopping, how the algorithm is structured efficiently in GLSL using separable passes, and key optimizations for real-time performance.

The Cross-Bilateral Filtering Algorithm

The primary algorithm used to evaluate depth-aware smoothing for ambient occlusion is the Joint Bilateral Filter (commonly referred to as the Cross-Bilateral Filter).

Standard bilateral filtering computes spatial weights based on Euclidean distance between pixels and photometric weights based on intensity differences in the input image. In noisy ambient occlusion textures, computing photometric weights directly on the noisy data causes filter failure. The cross-bilateral filter solves this by evaluating kernel weights using clean, high-precision guide buffers: the linear depth buffer and surface normal buffer from the G-buffer.

Weighting Function Formulation

The filter evaluates the output ambient occlusion value \(AO(p)\) for pixel \(p\) over a neighborhood \(\Omega\) using the equation:

\[AO(p) = \frac{\sum_{q \in \Omega} W(p, q) \cdot AO(q)}{\sum_{q \in \Omega} W(p, q)}\]

The total weight \(W(p, q)\) is the product of three distinct kernels:

  1. Spatial Weight (\(W_s\)): A standard 1D or 2D Gaussian kernel based on pixel distance \(\|p - q\|\).
  2. Depth Weight (\(W_d\)): An edge-stopping function measuring the linear depth difference: \[W_d(p, q) = \exp\left(-\frac{|Depth(p) - Depth(q)|}{\sigma_d \cdot |\nabla Depth(p)| + \epsilon}\right)\]
  3. Normal Weight (\(W_n\)): A geometric weight based on the angular difference between surface normals: \[W_n(p, q) = \max(0.0, \vec{N}_p \cdot \vec{N}_q)^{\sigma_n}\]

Separable Implementation in GLSL

Evaluating a 2D cross-bilateral filter over a wide radius is computationally expensive (\(O(K^2)\) per pixel). In modern engines, the algorithm is evaluated as a two-pass separable filter (horizontal blur followed by vertical blur), reducing complexity to \(O(2K)\). While cross-bilateral filters are mathematically non-separable due to the depth-dependent weights, the visual error is negligible compared to the performance gain.

Core GLSL Shader Pass

Below is a GLSL fragment shader implementation of a 1D horizontal pass for a depth-aware cross-bilateral filter.

#version 450 core

in vec2 vTexCoord;
out float FragColor;

uniform sampler2D uAOTex;
uniform sampler2D uLinearDepthTex;
uniform sampler2D uNormalTex;

uniform vec2 uTexelOffset; // e.g., vec2(1.0 / width, 0.0)
uniform float uDepthThreshold; // Scale factor for depth sensitivity

const int RADIUS = 4;
const float spatialWeights[5] = float[](0.227027, 0.1945946, 0.1216216, 0.054054, 0.016216);

void main() {
    float centerAO = texture(uAOTex, vTexCoord).r;
    float centerDepth = texture(uLinearDepthTex, vTexCoord).r;
    vec3 centerNormal = normalize(texture(uNormalTex, vTexCoord).xyz * 2.0 - 1.0);

    float totalWeight = spatialWeights[0];
    float totalAO = centerAO * totalWeight;

    for (int i = 1; i <= RADIUS; ++i) {
        float wSpatial = spatialWeights[i];

        // Sample both positive and negative offsets along the axis
        for (int dir = -1; dir <= 1; dir += 2) {
            vec2 offsetCoord = vTexCoord + float(i * dir) * uTexelOffset;

            float sampleAO = texture(uAOTex, offsetCoord).r;
            float sampleDepth = texture(uLinearDepthTex, offsetCoord).r;
            vec3 sampleNormal = normalize(texture(uNormalTex, offsetCoord).xyz * 2.0 - 1.0);

            // Calculate depth-aware difference
            float depthDiff = abs(centerDepth - sampleDepth);
            float wDepth = exp(-depthDiff * uDepthThreshold);

            // Calculate normal difference
            float wNormal = pow(max(dot(centerNormal, sampleNormal), 0.0), 16.0);

            // Combined edge-preserving weight
            float weight = wSpatial * wDepth * wNormal;

            totalAO += sampleAO * weight;
            totalWeight += weight;
        }
    }

    FragColor = totalAO / max(totalWeight, 0.0001);
}

Depth-Aware Edge Adjustments

Using a static depth threshold across the entire view frustum leads to over-blurring in the foreground and under-blurring in the distance. To mitigate this:

  • Linearize Depth: Use view-space linear depth rather than non-linear projection depth (\(Z_{buffer}\)). Linear depth ensures consistent metric units.
  • Scale with Depth Derivative: Scale \(\sigma_d\) by screen-space partial derivatives (dFdx and dFdy) of depth. Surfaces viewed at steep grazing angles naturally possess large depth deltas across adjacent pixels; scaling against surface derivatives prevents the filter from mistakenly identifying flat, angled surfaces as geometric discontinuities.

Multi-Scale Alternative: The À-Trous Wavelet Filter

When larger blur footprints are needed to smooth extreme low-frequency noise (such as 1-spp ambient occlusion), the standard bilateral kernel requires too many texture lookups. In such cases, the Edge-Avoiding À-Trous Filter is employed.

Instead of sampling every adjacent texel, the À-Trous algorithm evaluates the cross-bilateral weighting across multiple successive passes with increasing step sizes (\(2^k\) stride: 1, 2, 4, 8 pixels). This achieves a wide reconstruction footprint with a constant 5x5 sample kernel per pass, maintaining strict depth and normal edge-awareness while keeping GPU execution time predictable.