Banded Tone Math for Cel-Shaded GLSL Shaders

Cel-shading, or toon shading, relies on discretizing continuous illumination into distinct, quantized bands of light and shadow. This article outlines the mathematical formulations used to achieve banded tone transitions in GLSL shaders, covering direct linear quantization, threshold-based step functions, smoothed transitions, and 1D lookup textures.

The Lighting Foundation

Banded shading begins with standard Lambertian reflectance. Given a normalized surface normal vector \(\mathbf{N}\) and a normalized light direction vector \(\mathbf{L}\), the standard diffuse scalar intensity \(I\) is calculated as the dot product:

\[I = \max(\mathbf{N} \cdot \mathbf{L}, 0.0)\]

Alternatively, the Half-Lambert formulation can be used to prevent dark areas from losing detail by remapping the dot product from \([-1, 1]\) to \([0, 1]\):

\[I_{\text{half}} = 0.5(\mathbf{N} \cdot \mathbf{L}) + 0.5\]

Uniform Quantization via the Floor Function

To divide the continuous intensity \(I \in [0, 1]\) into \(k\) discrete, uniform tonal bands, a floor-based mathematical mapping is applied:

\[I_{\text{quantized}} = \frac{\lfloor I \cdot k \rfloor}{k - 1}\]

In GLSL, this is expressed directly in the fragment shader:

float NdotL = max(dot(normal, lightDir), 0.0);
float bands = 4.0;
float celIntensity = floor(NdotL * bands) / (bands - 1.0);

This formulation maps continuous illumination into evenly spaced steps between completely unlit (\(0.0\)) and fully lit (\(1.0\)).

Arbitrary Thresholds Using Step Functions

When artistic direction requires non-uniform band widths (for example, a large midtone region with a narrow shadow transition), the step function is employed:

\[I_{\text{cel}} = \sum_{i=1}^{m} w_i \cdot H(I - t_i)\]

Where \(H(x)\) is the Heaviside step function, \(t_i\) represents the transition threshold for band \(i\), and \(w_i\) represents the intensity weight added at that threshold. In GLSL, this uses the native step() function:

float NdotL = max(dot(normal, lightDir), 0.0);

// Define thresholds and step transitions
float celIntensity = 0.2; // Ambient/base shadow tone
celIntensity += step(0.3, NdotL) * 0.4; // Midtone band
celIntensity += step(0.7, NdotL) * 0.4; // Highlight band

Anti-Aliased and Soft Transitions Using Smoothstep

Hard step functions introduce high-frequency edges prone to geometric aliasing. To soften the transition or make band boundaries resolution-independent, the mathematical transition can be formulated using Hermite interpolation:

\[S(x, a, b) = \text{clamp}\left(\frac{x - a}{b - a}, 0.0, 1.0\right)^2 \cdot \left(3 - 2 \cdot \frac{x - a}{b - a}\right)\]

In GLSL, smoothstep creates a localized feathering window of width \(2\epsilon\) centered at threshold \(t\):

float NdotL = max(dot(normal, lightDir), 0.0);
float threshold = 0.4;
float epsilon = fwidth(NdotL) * 1.5; // Screen-space derivative anti-aliasing

float celIntensity = smoothstep(threshold - epsilon, threshold + epsilon, NdotL);

Using fwidth(NdotL) computes screen-space derivatives to scale the transition band dynamically, providing hardware-assisted anti-aliasing along tone edges.

Lookup Table (LUT) Mapping

For complex gradient curves or multi-color banding, the math is reduced to a 1D coordinate mapping:

\[u = \text{clamp}(I, 0.0, 1.0)\]

The color value is then fetched using a 1D ramp texture, shifting tone distribution math from dynamic ALU operations to texture interpolation:

float NdotL = clamp(dot(normal, lightDir), 0.0, 1.0);
vec3 celColor = texture(rampTexture, vec2(NdotL, 0.5)).rgb;