FXAA Edge Detection Implementation in GLSL

This article provides a practical guide to implementing the edge detection stage of Fast Approximate Anti-Aliasing (FXAA) in OpenGL Shading Language (GLSL). FXAA is a screen-space technique that smooths jagged silhouettes by analyzing local color contrast rather than evaluating geometric primitives. Readers will learn how FXAA converts color data to luminance, samples orthogonal neighbor pixels, evaluates local contrast against relative and absolute thresholds, and determines whether an edge is oriented horizontally or vertically.

The Principles of FXAA Edge Detection

FXAA detects edges based on human perceptual brightness, known as luminance, rather than raw RGB differences or depth buffer discontinuities. By processing luminance values of a target pixel alongside its immediate neighbors, the shader calculates the local contrast range. If this contrast exceeds predetermined thresholds, the pixel is flagged as an edge that requires blending.

1. Calculating Perceived Luminance

Luminance (\(L\)) is typically derived using standard colorimetric conversion coefficients that reflect human eye sensitivity to green wavelengths:

\[L = 0.299 \cdot R + 0.587 \cdot G + 0.114 \cdot B\]

In optimized rendering pipelines, luminance can be pre-calculated and stored inside the alpha channel of the primary render target to avoid redundant computations during the post-processing pass.

2. Gathering Local Neighborhood Samples

To detect an edge, the fragment shader samples the center pixel and its four direct orthogonal neighbors: North, South, East, and West.

vec2 texCoord = ...; // Normalized screen coordinates [0, 1]
vec2 inverseScreenSize = vec2(1.0 / screenWidth, 1.0 / screenHeight);

// Sample center and cardinal directions
float lumaCenter = rgb2luma(texture(screenTexture, texCoord).rgb);
float lumaDown   = rgb2luma(texture(screenTexture, texCoord + vec2(0.0, -1.0) * inverseScreenSize).rgb);
float lumaUp     = rgb2luma(texture(screenTexture, texCoord + vec2(0.0,  1.0) * inverseScreenSize).rgb);
float lumaLeft   = rgb2luma(texture(screenTexture, texCoord + vec2(-1.0, 0.0) * inverseScreenSize).rgb);
float lumaRight  = rgb2luma(texture(screenTexture, texCoord + vec2( 1.0, 0.0) * inverseScreenSize).rgb);

3. Contrast Evaluation and Thresholding

The algorithm determines the minimum and maximum luminance values among the sampled pixels to calculate the local contrast range:

float lumaMin = min(lumaCenter, min(min(lumaDown, lumaUp), min(lumaLeft, lumaRight)));
float lumaMax = max(lumaCenter, max(max(lumaDown, lumaUp), max(lumaLeft, lumaRight)));
float lumaRange = lumaMax - lumaMin;

To avoid processing flat surfaces or areas hidden in deep shadows, the local range is evaluated against two thresholds:

Relative Threshold (EDGE_THRESHOLD): Trims edges inside low-contrast areas (typically tuned between 0.125 and 0.250).
Absolute Threshold (EDGE_THRESHOLD_MIN): Prevents edge detection in near-black regions where aliasing is imperceptible (typically tuned between 0.0312 and 0.0833).

#define FXAA_EDGE_THRESHOLD 0.166
#define FXAA_EDGE_THRESHOLD_MIN 0.0833

if (lumaRange < max(FXAA_EDGE_THRESHOLD_MIN, lumaMax * FXAA_EDGE_THRESHOLD)) {
    // No anti-aliasing needed; return center color directly
    discard; 
}

Complete GLSL Edge Detection and Direction Pass

Once a pixel passes the threshold check, the algorithm samples diagonal neighbors (North-West, North-East, South-West, South-East) to compute directional gradients. This step establishes whether the edge runs predominantly along the horizontal or vertical axis.

#version 330 core

in vec2 vTexCoord;
out vec4 FragColor;

uniform sampler2D screenTexture;
uniform vec2 uInverseScreenSize;

#define FXAA_EDGE_THRESHOLD 0.166
#define FXAA_EDGE_THRESHOLD_MIN 0.0625

float rgb2luma(vec3 rgb) {
    return dot(rgb, vec3(0.299, 0.587, 0.114));
}

void main() {
    // Center luminance
    vec3 colorCenter = texture(screenTexture, vTexCoord).rgb;
    float lumaCenter = rgb2luma(colorCenter);

    // Cardinal neighbor luminance
    float lumaDown  = rgb2luma(texture(screenTexture, vTexCoord + vec2(0.0, -1.0) * uInverseScreenSize).rgb);
    float lumaUp    = rgb2luma(texture(screenTexture, vTexCoord + vec2(0.0,  1.0) * uInverseScreenSize).rgb);
    float lumaLeft  = rgb2luma(texture(screenTexture, vTexCoord + vec2(-1.0, 0.0) * uInverseScreenSize).rgb);
    float lumaRight = rgb2luma(texture(screenTexture, vTexCoord + vec2( 1.0, 0.0) * uInverseScreenSize).rgb);

    // Compute range and evaluate thresholds
    float lumaMin = min(lumaCenter, min(min(lumaDown, lumaUp), min(lumaLeft, lumaRight)));
    float lumaMax = max(lumaCenter, max(max(lumaDown, lumaUp), max(lumaLeft, lumaRight)));
    float lumaRange = lumaMax - lumaMin;

    if (lumaRange < max(FXAA_EDGE_THRESHOLD_MIN, lumaMax * FXAA_EDGE_THRESHOLD)) {
        FragColor = vec4(colorCenter, 1.0);
        return;
    }

    // Corner luminance for orientation determination
    float lumaDownLeft  = rgb2luma(texture(screenTexture, vTexCoord + vec2(-1.0, -1.0) * uInverseScreenSize).rgb);
    float lumaUpRight   = rgb2luma(texture(screenTexture, vTexCoord + vec2( 1.0,  1.0) * uInverseScreenSize).rgb);
    float lumaUpLeft    = rgb2luma(texture(screenTexture, vTexCoord + vec2(-1.0,  1.0) * uInverseScreenSize).rgb);
    float lumaDownRight = rgb2luma(texture(screenTexture, vTexCoord + vec2( 1.0, -1.0) * uInverseScreenSize).rgb);

    // Combine cardinal and diagonal deltas to measure edge orientation
    float lumaDownUp    = lumaDown + lumaUp;
    float lumaLeftRight = lumaLeft + lumaRight;

    float lumaLeftCorners  = lumaDownLeft + lumaUpLeft;
    float lumaDownCorners  = lumaDownLeft + lumaDownRight;
    float lumaRightCorners = lumaDownRight + lumaUpRight;
    float lumaUpCorners    = lumaUpRight + lumaUpLeft;

    // Gradient calculations
    float edgeHorizontal = abs(-2.0 * lumaLeft + lumaLeftCorners)  +
                           abs(-2.0 * lumaCenter + lumaDownUp) * 2.0 +
                           abs(-2.0 * lumaRight + lumaRightCorners);

    float edgeVertical   = abs(-2.0 * lumaUp + lumaUpCorners)      +
                           abs(-2.0 * lumaCenter + lumaLeftRight) * 2.0 +
                           abs(-2.0 * lumaDown + lumaDownCorners);

    bool isHorizontal = edgeHorizontal >= edgeVertical;

    // Edge detected: output debug visualizer or proceed to the blending stage
    FragColor = isHorizontal ? vec4(1.0, 0.0, 0.0, 1.0) : vec4(0.0, 0.0, 1.0, 1.0);
}

The resulting boolean isHorizontal identifies the normal direction required for subsequent sub-pixel edge tracing and texture filtering passes across the detected boundary.