Stochastic Alpha Testing for Shadows in GLSL

This article explains how transparent shadow transmission maps are evaluated in real-time graphics using stochastic alpha testing in OpenGL Shading Language (GLSL). By converting continuous surface opacity and transmission values into probabilistic binary coverage via noise-driven discards, developers can leverage standard hardware depth maps to simulate soft, semi-transparent shadows without the immense memory overhead or sorting requirements of deep shadow maps.

The Mechanics of Stochastic Alpha Testing

Standard alpha testing discards fragments when an opacity value falls below a fixed threshold (e.g., \(\alpha < 0.5\)), resulting in hard, binary silhouettes unsuitable for soft, translucent materials. Stochastic alpha testing replaces this static cutoff with a random or pseudo-random threshold \(\xi \in [0, 1)\).

When rendering to a shadow map, a fragment with transparency \(\alpha\) (or transmission \(\tau = 1 - \alpha\)) is discarded if \(\alpha < \xi\). Over an aggregate area or across temporal frames, the probability that a fragment writes to the depth buffer is proportional to its opacity:

\[P(\text{Occlusion}) = \alpha\]

This effectively encodes semi-transparency into a high-frequency spatial dither pattern stored in a standard single-channel depth texture.

Shadow Map Generation in GLSL

During the shadow generation pass, transparent materials sample their diffuse/alpha or transmission map. Instead of writing complex blended depth representations, the fragment shader computes a deterministic noise value per pixel—typically using screen-space interleaved gradient noise or precomputed blue noise—and discards fragments conditionally.

#version 450 core

in vec2 vTexCoord;
in vec4 vClipPosition;

uniform sampler2D uTransmissionMap;
uniform sampler2D uBlueNoise;
uniform vec2 uScreenResolution;

void main() {
    // Sample transmission (0.0 = fully opaque, 1.0 = fully transparent)
    float transmission = texture(uTransmissionMap, vTexCoord).r;
    float opacity = 1.0 - transmission;

    // Fetch a pseudo-random value in [0, 1] using screen coordinates
    vec2 noiseCoord = gl_FragCoord.xy / vec2(textureSize(uBlueNoise, 0));
    float threshold = texture(uBlueNoise, noiseCoord).r;

    // Stochastic discard
    if (opacity < threshold) {
        discard;
    }

    // Depth is written automatically to the shadow map
}

Blue noise is strongly preferred over white noise or simple hash functions because its energy is concentrated in high spatial frequencies, avoiding low-frequency clumping and making subsequent filtering passes substantially cleaner.

Handling Colored Shadow Transmission

Translucent objects such as stained glass transmit tinted light. A standard single-channel depth shadow map cannot capture light attenuation across RGB wavelengths natively. To evaluate colored transmission maps stochastically, the shader samples a random color channel (wavelength) per shadow sample:

  1. A random number determines which channel to test: Red, Green, or Blue.
  2. The fragment evaluates stochastic alpha testing using the transmittance of the selected channel.
  3. In the lighting pass, separate shadow tests are conducted for each color channel or accumulated across temporal frames using stochastic spectral sampling.

Alternatively, shadow transmission can be split across multiple render targets or evaluated during the receiver pass using a low-resolution transmission buffer alongside the depth map.

Filtering and Reconstruction at the Receiver

Because the shadow map contains high-frequency binary noise, direct point-sampling on the receiver produces visual grain. The receiver evaluates the transmission by convolving the stochastic depth buffer:

  1. Percentage-Closer Filtering (PCF): The receiver samples the shadow map across a spatial kernel. The binary occlusion results are averaged, naturally reconstructing the continuous transmission factor without custom shadow-blending algorithms.
  2. Temporal Anti-Aliasing (TAA): By jittering the blue noise pattern in the shadow map pass each frame, the receiver's spatial filter combines with temporal accumulation, converging the noisy dither into smooth, noise-free transparent shadows.