DXR Recursive Ray Tracing: Threading and Registers

Recursive ray generation in DirectX Raytracing (DXR) allows shaders—such as Closest Hit shaders—to spawn secondary rays via nested TraceRay() calls, creating dynamic rendering paths for reflections, refractions, and global illumination. While flexible, this approach introduces substantial performance bottlenecks at the hardware level. This article examines how nested ray recursion inflates GPU register allocation, triggers register spilling to slow memory, causes thread divergence, and severely degrades hardware thread scheduling efficiency.

GPU Register Pressure and the Virtual Call Stack

Traditional GPU architectures rely on thousands of lightweight threads running in lockstep (Wavefronts on AMD, Warps on NVIDIA). Each thread relies on a shared pool of general-purpose registers (GPRs) to maintain its state. Unlike standard CPU execution, modern GPUs lack a dedicated hardware call stack designed for deep, dynamic recursion.

When a shader invokes TraceRay() recursively, the execution state of the parent shader must be preserved until the child ray traversal and its subsequent shading complete. This creates several acute hardware impacts:

  • Inflated Live Variable Lifetimes: All local variables needed after the nested TraceRay() call must remain allocated throughout the entire traversal of the child ray. This artificially lengthens variable lifespans, dramatically increasing the total register count required per thread.
  • Reduced Occupancy: Hardware thread schedulers allocate execution slots based on resource availability. As the number of registers allocated per thread increases, the total number of active warps or wavefronts that can reside concurrently on a Streaming Multiprocessor (SM) or Compute Unit (CU) drops. This reduction in thread occupancy limits the GPU's primary mechanism for hiding memory latency.
  • Register Spilling to Local Memory: When recursive depth requires more registers than the hardware allocator can provide, the compiler must spill the excess state into local thread storage (scratch memory located in VRAM or L2 cache). Reading and writing stack frames to memory turns low-latency register operations into high-latency memory transactions, multiplying memory bandwidth saturation.

Impact on Thread Scheduling and Execution Coherence

GPU schedulers depend on uniform execution across lanes within a warp or wavefront. Recursive ray dispatching fundamentally breaks this uniformity through multiple mechanisms:

  • Execution and Path Divergence: In any single wave of 32 or 64 threads, individual rays will hit different surfaces, enter different hit shaders, or terminate early (e.g., hitting a light source or reaching the skybox). If some threads recurse while others finish, the GPU scheduler must serialize execution paths. Threads that have completed must idle while active lanes finish their deeper recursion, slashing SIMD/SIMT hardware utilization.
  • Unpredictable Latency and Warp Stalls: Traversal of nested rays through the Bounding Volume Hierarchy (BVH) has unpredictable duration. Because lower occupancy leaves the scheduler with fewer alternative warps to execute during stalls, execution pipelines sit idle waiting for ray-box and ray-triangle intersection hardware to resolve.
  • Dynamic State Management Overhead: The DXR runtime must dynamically manage ray payloads across recursion boundaries. Passing large ray payload structures down recursive chains demands additional memory overhead, which further fragments the cache hierarchy and slows scheduler dispatch cycles.

Mitigation Strategies in Modern DirectX

To counter the scheduling stalls and high register pressure of recursion, modern DirectX pipelines generally avoid deep TraceRay() nesting in favor of iterative paradigms:

  • Loop-Based Ray Generation: Rather than firing rays from hit shaders, developers maintain an iterative loop within the primary Ray Generation Shader. Shaders write hit surface parameters into an output buffer, allowing the main loop to cast subsequent rays sequentially, reducing register lifetimes to a single ray's scope.
  • Inline Ray Tracing (RayQuery in DXR 1.1): Inline ray queries grant explicit control over the traversal process directly within compute or pixel shaders, eliminating the runtime hit-shader dispatch table entirely and keeping the execution stack flat.
  • Payload Trimming: Restricting the Ray Payload to the absolute minimum byte count reduces the live state carried across recursive boundaries, minimizing spilled memory when recursion cannot be completely avoided.