Preventing Stack Overflow in GLSL Ray Tracing

In GLSL ray tracing pipelines, managing recursive ray depth is critical because graphics hardware lacks a traditional dynamic execution stack, making true recursion either strictly prohibited or functionally dangerous. This article examines how developers handle ray bounces by converting recursive algorithms into iterative loops, managing ray state manually, applying path-termination techniques like Russian roulette, and utilizing the explicit depth configurations provided by modern hardware ray tracing extensions.

The Hardware Recursion Limitation

Graphics Processing Units (GPUs) rely on SIMD/SIMT architectures optimized for predictable, uniform memory allocation across thousands of parallel threads. Traditional recursion requires pushing dynamic stack frames containing return pointers, local variables, and shader state onto a call stack. Because GLSL fragment and compute shaders allocate a fixed amount of register space per thread at compile time, unbounded or deep recursion leads to register spills, severe performance degradation, or shader compilation errors.

Iterative Ray Unrolling

To circumvent the lack of a dynamic stack, developers flatten recursive functions like Whitted-style ray tracing or path tracing into iterative for or while loops.

Instead of a function recursively calling itself upon hitting a surface, a single ray's lifecycle is tracked linearly across a predefined number of iterations:

vec3 throughput = vec3(1.0);
vec3 accumulatedColor = vec3(0.0);
Ray currentRay = initialRay;

for (int bounce = 0; bounce < MAX_BOUNCES; ++bounce) {
    HitRecord hit = intersectScene(currentRay);
    
    if (!hit.hit) {
        accumulatedColor += throughput * sampleEnvironment(currentRay);
        break;
    }

    accumulatedColor += throughput * hit.emission;
    throughput *= hit.albedo;

    currentRay.origin = hit.position + hit.normal * 0.001;
    currentRay.direction = reflect(currentRay.direction, hit.normal);
}

In this model, the state (origin, direction, and color throughput) is updated in place, completely eliminating the need to preserve prior frames on a stack.

Managing Branching Rays with Emulated Stacks

When an algorithm requires branching—such as evaluating both reflection and refraction at a dielectric surface—a single path loop is insufficient. In standard GLSL compute or fragment stages, this is managed by allocating a fixed-size array in local memory to emulate a Last-In, First-Out (LIFO) stack:

#define STACK_SIZE 16

Ray rayStack[STACK_SIZE];
int stackPointer = 0;

// Push
rayStack[stackPointer++] = reflectedRay;

// Pop
Ray nextRay = rayStack[--stackPointer];

The stack pointer is explicitly bounds-checked. If the pointer reaches STACK_SIZE, further branching is clamped, discarding extra rays to guarantee the memory boundary is never breached.

Probabilistic Early Termination: Russian Roulette

Even with a fixed maximum depth, simulating multiple bounces per pixel significantly increases execution time and thread divergence. To prevent deep, computationally expensive paths that contribute negligibly to the final image, pipelines employ Russian roulette.

After a minimum number of bounces, the continuation of a ray is determined probabilistically based on its current throughput:

if (bounce > MIN_BOUNCES) {
    float p = max(throughput.r, max(throughput.g, throughput.b));
    if (randomFloat() > p) {
        break; // Terminate ray early
    }
    throughput /= p; // Boost remaining energy to preserve unbiased rendering
}

This terminates low-energy paths early, keeping the active depth shallow across the majority of threads.

Hardware Ray Tracing Extensions (GL_EXT_ray_tracing)

In modern pipelines using hardware-accelerated ray tracing (via Vulkan or KHR extensions in GLSL), recursion can be declared using traceRayEXT(). However, the API enforces a strict recursion budget configured via maxRayRecursionDepth in the pipeline properties.

Because pipeline recursion depth consumes large amounts of fast memory, modern best practices dictate setting maxRayRecursionDepth to 1 (or 2 for simple shadow queries). The actual ray traversal is then managed within the Ray Generation shader using the same iterative loop approach, passing data via ray payloads rather than nesting traceRayEXT() calls inside Closest Hit shaders.