Screen-Space Reflection Edge Fade in GLSL

Screen-space reflections (SSR) produce harsh visual cutoffs when reflection rays traverse beyond the visible viewport borders. Implementing boundary vignette attenuation solves this issue by computing an attenuation factor based on the distance between the projected sample UVs and the screen boundaries, smoothly fading the reflection color to zero as it approaches the viewport perimeter.

The Mathematical Formulas

There are two primary mathematical approaches used in GLSL to calculate screen-space boundary attenuation:

1. Distance-to-Edge with Smoothstep (Separable)

This approach computes the linear distance to the nearest edge on each axis independently, applies a Hermite interpolation (smoothstep) across an edge threshold, and multiplies the axes together.

Given UV coordinates \(\mathbf{uv} \in [0, 1]^2\) and a fade margin \(m \in (0, 0.5]\):

\[\mathbf{d} = \min(\mathbf{uv}, 1.0 - \mathbf{uv})\] \[\mathbf{fade} = \text{smoothstep}(0.0, m, \mathbf{d})\] \[\text{attenuation} = \mathbf{fade}_x \cdot \mathbf{fade}_y\]

2. Normalized Device Coordinate (NDC) Chebyshev Metric

Alternatively, you can transform the UVs into normalized coordinates centered at the origin \([-1, 1]\), compute the maximum absolute deviation (Chebyshev distance), and fade out toward the outer edge.

Given UV coordinates \(\mathbf{uv} \in [0, 1]^2\) and a fade start threshold \(t \in [0, 1)\):

\[\mathbf{ndc} = |\mathbf{uv} \cdot 2.0 - 1.0|\] \[d_{\max} = \max(\mathbf{ndc}_x, \mathbf{ndc}_y)\] \[\text{attenuation} = 1.0 - \text{smoothstep}(t, 1.0, d_{\max})\]

GLSL Implementations

Method 1: Separable Edge Attenuation

This function calculates smooth fading individually on both the X and Y axes, producing rounded corner falloffs.

/**
 * Calculates vignette attenuation for screen edges.
 * @param uv Normalized screen-space texture coordinates in [0, 1].
 * @param edgeMargin The thickness of the fade zone (typically 0.05 to 0.2).
 * @return Attenuation factor in the range [0.0, 1.0].
 */
float calculateSSREdgeFade(vec2 uv, float edgeMargin) {
    vec2 edgeDist = min(uv, vec2(1.0) - uv);
    vec2 fade = smoothstep(vec2(0.0), vec2(edgeMargin), edgeDist);
    return fade.x * fade.y;
}

Method 2: Center-Offset Maximum Bound Fade

This method measures the distance from the center of the viewport to the nearest outer edge using normalized device coordinates.

/**
 * Calculates vignette attenuation based on maximum offset from the screen center.
 * @param uv Normalized screen-space texture coordinates in [0, 1].
 * @param fadeStart Normalized distance from center where fade begins [0.0, 1.0].
 * @return Attenuation factor in the range [0.0, 1.0].
 */
float calculateSSREdgeFadeChebyshev(vec2 uv, float fadeStart) {
    vec2 ndc = abs(uv * 2.0 - 1.0);
    float maxDist = max(ndc.x, ndc.y);
    return clamp(1.0 - smoothstep(fadeStart, 1.0, maxDist), 0.0, 1.0);
}

Applying the Vignette to Reflections

In an SSR shader pipeline, the attenuation factor scales the incoming reflected radiance before compositing with the scene color:

vec2 hitUV = rayMarchResult.xy;
bool hit = rayMarchResult.z > 0.0;

if (hit) {
    vec3 reflectionColor = texture(sceneColorTexture, hitUV).rgb;
    
    // Compute boundary fade
    float edgeFactor = calculateSSREdgeFade(hitUV, 0.1);
    
    // Combine with other attenuation factors (e.g., ray distance, roughness)
    float finalAlpha = edgeFactor * hitConfidence;
    
    outColor = mix(baseColor, reflectionColor, finalAlpha);
}

This ensures that any reflection ray intersecting scene geometry near the edges of the frame diminishes smoothly without popping or displaying clipping artifacts.