How GLSL Detects Total Internal Reflection

In GLSL shaders, total internal reflection (TIR) occurs during refraction calculations when light travels from a medium with a higher refractive index to one with a lower refractive index at an angle exceeding the critical angle. Rather than throwing an error or halting execution, GLSL evaluates the underlying mathematics of Snell's Law and flags total internal reflection by returning a zero vector (vec3(0.0)) from the built-in refract function. Shaders detect this condition either by checking for a zero-length result or by manually calculating the refraction discriminant before computing the refracted vector.

The Mathematics Behind GLSL Refraction

The built-in GLSL function refract(I, N, eta) calculates the transmitted vector based on Snell's Law:

\[n_1 \sin(\theta_1) = n_2 \sin(\theta_2)\]

Here, I is the normalized incident ray, N is the normalized surface normal, and eta (\(\eta\)) represents the ratio of the indices of refraction (\(n_1 / n_2\)).

When translating this into vector mathematics, the direction of the refracted ray involves computing a term under a square root (the discriminant \(k\)):

\[k = 1.0 - \eta^2 \cdot (1.0 - (\vec{N} \cdot \vec{I})^2)\]

Since \(\cos^2(\theta_1) = (\vec{N} \cdot \vec{I})^2\) and \(\sin^2(\theta_1) = 1.0 - \cos^2(\theta_1)\), this equation determines whether \(\sin(\theta_2)\) remains valid:

\[k = 1.0 - \eta^2 \sin^2(\theta_1)\]

If light travels into an optically less dense medium (\(\eta > 1.0\)) and the incident angle \(\theta_1\) exceeds the critical angle, \(\eta^2 \sin^2(\theta_1)\) becomes greater than \(1.0\). Consequently, \(k\) becomes negative. Because taking the square root of a negative number produces an imaginary number, no physical refraction can occur.

Detection via the Built-in refract Function

According to the OpenGL Shading Language specification, if \(k < 0.0\), the refract function returns a vector where all components are set to 0.0. A shader detects total internal reflection by inspecting this return value:

vec3 I = normalize(incidentRay);
vec3 N = normalize(surfaceNormal);
float eta = n1 / n2;

vec3 R = refract(I, N, eta);

if (dot(R, R) == 0.0) {
    // Total Internal Reflection detected:
    // Fall back to pure reflection
    R = reflect(I, N);
}

Using dot(R, R) == 0.0 or length(R) == 0.0 reliably confirms whether refraction failed due to total internal reflection.

Manual Discriminant Detection

In high-performance rendering or complex physically-based shading models, developers often calculate the discriminant \(k\) manually. Calculating \(k\) directly avoids computing the full refraction vector when TIR occurs and allows conditional branching or smooth blending between reflection and refraction:

float cosTheta = dot(-I, N);
float k = 1.0 - eta * eta * (1.0 - cosTheta * cosTheta);

if (k < 0.0) {
    // Total Internal Reflection occurs
    color = calculateReflection(reflect(I, N));
} else {
    // Standard refraction occurs
    vec3 refractedRay = eta * I + (eta * cosTheta - sqrt(k)) * N;
    color = calculateRefraction(refractedRay);
}

By verifying whether \(k < 0.0\) or checking if refract() returns vec3(0.0), GLSL shaders accurately isolate total internal reflection and switch from refraction rendering to standard specular reflection.