Bayer Matrix Dithering in GLSL for Color Banding

Color banding occurs when smooth color gradients are quantized into discrete steps due to limited color bit depth, creating harsh, visible lines across a render. In real-time rendering using GLSL (OpenGL Shading Language), dithering algorithms such as Bayer matrix ordered dithering eliminate these artifacts by dispersing quantization errors across neighboring pixels. By adding a small, deterministic threshold value to fragment colors before quantization, the shader leverages human visual perception to blend discrete color boundaries into what appears to be a seamless, high-precision gradient.

The Cause of Color Banding

Digital displays and standard render buffers typically operate at 8 bits per channel (256 shades per color component). When rendering subtle lighting, atmospheric fog, or wide gradients across large screen areas, the difference between adjacent mathematical color values is often smaller than the minimum representable step size (\(1/255\)). Consequently, continuous values are truncated or rounded to the nearest integer, forming visually jarring bands of uniform color instead of smooth transitions.

How Dithering Solves Banding

Dithering introduces controlled high-frequency noise into the image prior to quantization. Instead of a hard transition where every pixel shifts from shade \(N\) to \(N+1\) simultaneously, dithering causes pixels near the threshold to intersperse both values.

The human visual system naturally performs low-pass spatial filtering, averaging adjacent pixels at a distance. As a result, an area with an alternating 50/50 mix of shade \(N\) and shade \(N+1\) is perceived as an intermediate tone (\(N + 0.5\)), effectively simulating higher bit depths without increasing the frame buffer's memory footprint.

The Bayer Matrix in GLSL

While error-diffusion algorithms (like Floyd-Steinberg) propagate quantization errors to neighboring pixels, they are inherently sequential and unsuitable for the parallel architecture of GPUs. GLSL shaders process fragments simultaneously and independently, making ordered dithering via a Bayer matrix the industry standard for real-time graphics.

The Bayer matrix is an \(N \times N\) grid of normalized threshold values (commonly \(2 \times 2\), \(4 \times 4\), or \(8 \times 8\)) defined recursively to maximize the spatial distance between similar values. This spatial distribution minimizes low-frequency clumping, producing a subtle cross-hatch pattern that the human eye easily blends.

Implementation in a Fragment Shader

In a GLSL fragment shader, ordered dithering is implemented directly on the final output color using the fragment's screen coordinates:

  1. Coordinate Mapping: The fragment position (gl_FragCoord.xy) is mapped into the Bayer matrix dimensions using the modulo operator to tile the pattern seamlessly across the screen.
  2. Threshold Lookup: The matrix yields a static threshold value, typically normalized to a range such as \([0, 1)\) or \([-0.5, 0.5)\).
  3. Offset Application: The threshold is scaled by the quantization step (for 8-bit color, this is roughly \(1.0 / 255.0\)) and added to the continuous color value.
  4. Quantization: The altered color is written to the output buffer, where standard hardware rounding converts the offset values into the interleaved pixel pattern.
// Example of a 4x4 Bayer matrix lookup in GLSL
float bayer4x4(vec2 uv) {
    int x = int(mod(uv.x, 4.0));
    int y = int(mod(uv.y, 4.0));
    
    int index = x + y * 4;
    float m[16] = float[16](
         0.0,  8.0,  2.0, 10.0,
        12.0,  4.0, 14.0,  6.0,
         3.0, 11.0,  1.0,  9.0,
        15.0,  7.0, 13.0,  5.0
    );
    
    return (m[index] / 16.0) - 0.5;
}

void main() {
    vec3 linearColor = calculateColor(); // Smooth, banded gradient
    
    // Scale dither by quantization step
    float dither = bayer4x4(gl_FragCoord.xy) * (1.0 / 255.0);
    
    gl_FragColor = vec4(linearColor + dither, 1.0);
}

Performance and Visual Characteristics

Bayer matrix dithering delivers an optimal trade-off between performance and visual quality. Because the operation relies solely on simple arithmetic and screen-space coordinates without dynamic texture fetches, the computational cost is virtually negligible. While random noise dithering can cause temporal grain or "crawling" when the camera moves, ordered Bayer dithering remains visually stable relative to screen space, providing a smooth and reliable remedy for gradient banding in real-time GLSL applications.