How Does the isnan Function Detect Float Errors in GLSL?

The isnan built-in function in GLSL provides a standardized mechanism for identifying "Not-a-Number" (NaN) floating-point states generated by invalid mathematical operations within shader programs. By inspecting the bitwise representation of floating-point values according to IEEE 754 conventions—or leveraging specialized GPU hardware comparison flags—isnan allows shaders to catch undefined numerical results before they propagate through the rendering pipeline, causing visual artifacts or corrupting buffer outputs.

The Nature of Floating-Point Errors in Shaders

Modern GPUs perform floating-point arithmetic following standard conventions derived from IEEE 754 single-precision float standards. During shader execution, certain mathematical operations produce results that are mathematically undefined or unrepresentable:

• Division by zero: \(0.0 / 0.0\)
• Invalid square roots: \(\sqrt{x}\) where \(x < 0.0\)
• Indeterminate forms: \(\infty - \infty\) or \(0.0 \times \infty\)
• Out-of-range inverses: \(\text{asin}(x)\) or \(\text{acos}(x)\) where \(\vert{}x\vert{} > 1.0\)
• Logarithms of non-positive numbers: \(\ln(x)\) where \(x \le 0.0\)

When a shader encounters one of these calculations, the floating-point unit generates a NaN bit pattern rather than a valid real number.

IEEE 754 Representation of NaN

In single-precision floating-point format (32-bit), a standard number is split into three parts: a 1-bit sign, an 8-bit biased exponent, and a 23-bit fraction (mantissa).

A value evaluates as NaN when:

• The exponent field contains all ones (0xFF or \(255\)).
• The mantissa (fraction) field is non-zero.

If the mantissa is entirely zero while the exponent is all ones, the representation indicates positive or negative infinity (\(\pm\infty\)). The isnan function distinguishes NaN specifically by checking that the exponent bits are saturated and at least one bit in the fraction is active.

Mechanism of Detection in GLSL

GLSL defines isnan as an overloaded function accepting float, vec2, vec3, or vec4 (as well as double-precision types when supported), returning a corresponding bool or bvec component-wise.

// Function signatures
bool isnan(float x);
bvec2 isnan(vec2 x);
bvec3 isnan(vec3 x);
bvec4 isnan(vec4 x);

Under the hood, GPU architectures detect NaN values using two primary methods:

1. Hardware IEEE 754 Comparison Logic

According to IEEE 754 rules, NaN has a unique property: it is unordered, meaning any comparison with NaN (including \(x == x\)) evaluates to false. While standard CPU code often tests (x != x) to detect NaN, GPU drivers often optimize away such comparisons when strict IEEE compliance is relaxed. The built-in isnan intrinsic bypasses compiler optimization ambiguities by compiling directly to specialized GPU instruction opcodes (such as test.nan or dedicated floating-point test instructions) that inspect the register state directly.

2. Bitwise Inspection

When strict floating-point conformance or native hardware instructions are absent, the shader compiler can lower isnan to equivalent bitwise operations using floatBitsToUint:

bool customIsNan(float val) {
    uint u = floatBitsToUint(val);
    return ((u & 0x7F800000u) == 0x7F800000u) && ((u & 0x007FFFFFu) != 0u);
}

This logic isolates the 8-bit exponent with the bitmask 0x7F800000u to check for saturation and verifies that the fractional payload in 0x007FFFFFu contains at least one set bit.

Practical Usage in Shader Code

Once NaN enters a computation, any further arithmetic involving that value also produces NaN. This "NaN poisoning" can corrupt lighting calculations, bloom passes, post-processing filters, or ray marching steps, frequently appearing on-screen as black pixels, white flashes, or missing geometry.

Using isnan allows shaders to sanitize inputs or provide fallback values:

vec3 safeColor(vec3 color, vec3 fallback) {
    bvec3 isInvalid = isnan(color);
    return mix(color, fallback, vec3(isInvalid));
}

In control flows where individual vector components might encounter singularities, component-wise branching or replacement using isnan preserves numerical stability across the entire render target.