Stylized Rim Lighting with Inverted Fresnel in GLSL

This article explains how to compute stylized rim lighting in GLSL shaders using an inverted Fresnel approach. By evaluating the relationship between a surface normal and the camera view vector, shaders can detect the outer silhouettes of 3D geometry. Inverting this dot product and applying shaping functions such as power curves or smooth thresholding allows developers to generate sharp anime-style outlines, glowing energy halos, or soft cartoon highlights suitable for non-photorealistic rendering (NPR).

The Core Concept: View Angle and Inversion

Standard rendering models calculate light behavior using vectors for the surface normal (\(\mathbf{N}\)) and the camera's viewing direction (\(\mathbf{V}\)). In world or view space, the dot product \(\mathbf{N} \cdot \mathbf{V}\) measures how directly a surface faces the observer:

  • \(\mathbf{N} \cdot \mathbf{V} \approx 1.0\): The surface directly faces the camera (center of the object).
  • \(\mathbf{N} \cdot \mathbf{V} \approx 0.0\): The surface faces perpendicular to the camera (grazing angles or outer edges).

A physical Fresnel effect reflects more light as the angle of incidence approaches grazing angles. In a shader, an inverted Fresnel rim calculation explicitly isolates these glancing angles by subtracting the dot product from 1.0:

\[\text{Rim Factor} = 1.0 - \max(\mathbf{N} \cdot \mathbf{V}, 0.0)\]

This operation shifts values so that geometry facing the camera receives a value of 0.0, while geometric silhouettes approach 1.0.

Stylizing the Falloff

Raw linear falloff creates a wide, hazy gradient across the mesh. To achieve a stylized, illustrative, or cel-shaded look, the rim value must be shaped mathematically before adding it to the final fragment output.

1. Exponential Falloff (pow)

To produce a concentrated edge highlight while suppressing brightness across the rest of the surface, apply a power function:

float rim = 1.0 - max(dot(normal, viewDir), 0.0);
rim = pow(rim, rimPower); // Higher values make the rim thinner

2. Stepped Cel-Shading (smoothstep or step)

For an anime or comic aesthetic with sharp edge cutoffs, use step functions instead of smooth gradients:

// Hard cel edge
float hardRim = step(0.6, rim);

// Softened cel edge
float softRim = smoothstep(0.5, 0.55, rim);

3. Light Masking

Stylized rim lights often only appear on the unlit or back-lit side of a character. This can be achieved by multiplying the rim factor by the directional light factor or an inverted Lambertian term:

float NdotL = max(dot(normal, lightDir), 0.0);
float backLightMask = 1.0 - NdotL;
float maskedRim = rim * backLightMask;

Complete GLSL Implementation

Below is a complete GLSL fragment shader implementation demonstrating the inverted Fresnel rim calculation with color blending and threshold controls.

#version 330 core

in vec3 vNormal;
in vec3 vWorldPos;

out vec4 FragColor;

uniform vec3 uCameraPos;
uniform vec3 uBaseColor;
uniform vec3 uRimColor;
uniform float uRimPower;      // Controls edge thickness (e.g., 3.0 to 5.0)
uniform float uRimIntensity;  // Brightness multiplier
uniform float uRimThreshold;  // Cutoff threshold for stylized edges

void main()
{
    // Ensure vectors are normalized
    vec3 N = normalize(vNormal);
    vec3 V = normalize(uCameraPos - vWorldPos);

    // Compute inverted Fresnel term
    float NdotV = max(dot(N, V), 0.0);
    float rim = 1.0 - NdotV;

    // Shape falloff using exponentiation
    rim = pow(rim, uRimPower);

    // Apply a threshold for a clean, stylized NPR border
    rim = smoothstep(uRimThreshold - 0.05, uRimThreshold + 0.05, rim);

    // Calculate final rim lighting contribution
    vec3 rimFinal = uRimColor * rim * uRimIntensity;

    // Combine with base surface color
    vec3 finalColor = uBaseColor + rimFinal;

    FragColor = vec4(finalColor, 1.0);
}

Performance and Coordinate Space Considerations

  • Coordinate Spaces: The normal vector and view vector must be in the same coordinate space. Calculating them in view (camera) space simplifies the calculation because the view vector is constantly vec3(0.0, 0.0, 1.0).
  • Normalization: Interpolation across triangles denormalizes vectors. Re-normalizing both \(\mathbf{N}\) and \(\mathbf{V}\) inside the fragment shader is required to prevent jagged, uneven light bands along the geometry.