How terminateRayEXT Works in GLSL Shadow Rays

In GLSL ray tracing pipelines using the GL_EXT_ray_tracing extension, the terminateRayEXT() function immediately halts the traversal of a ray once an intersection is confirmed. For shadow rays—which typically only need to determine binary visibility (whether a light source is occluded or not) rather than identifying the closest surface—calling terminateRayEXT() inside an Any-Hit shader stops the acceleration structure traversal instantly, skipping all remaining intersection tests to optimize rendering performance.

The Purpose of Shadow Rays

Unlike primary or reflection rays that search for the closest geometric surface to evaluate shading attributes, shadow rays (often called occlusion rays) merely ask whether any opaque geometry exists between the shaded point and a light source. Because locating the nearest surface along the ray is unnecessary for a simple shadow test, traversing the entire bounding volume hierarchy (BVH) to sort hits from front to back introduces avoidable GPU overhead.

How terminateRayEXT() Executes

When an Any-Hit shader executes, the GPU has encountered a potential intersection candidate. If the shader logic confirms the surface is opaque (for example, passing an alpha-mask or punch-through texture evaluation), calling terminateRayEXT() does the following:

  1. Accepts the Current Intersection: The current candidate is finalized as an accepted hit along the ray path.
  2. Aborts BVH Traversal: The hardware traversal unit ceases searching for any additional bounding boxes, primitives, or geometry instances along the ray.
  3. Discards Remaining Candidates: Any other potential intersections that were queued or pending evaluation are immediately rejected and abandoned.
  4. Returns Execution Flow: Traversal concludes, causing the pipeline to transition to the Closest-Hit shader (if bound) or directly back to the Ray Generation shader, returning an occluded state via the ray payload.

Interaction with Ray Flags

While Vulkan and GLSL offer the ray flag gl_RayFlagsTerminateOnFirstHitEXT to automatically terminate rays on opaque geometry, terminateRayEXT() is essential when handling non-opaque or alpha-tested materials. In these cases, geometry cannot be marked as fully opaque in the acceleration structure. The Any-Hit shader must first sample the material texture:

  • If the sample is transparent, the shader calls ignoreIntersectionEXT() to continue searching.
  • If the sample is opaque, the shader calls terminateRayEXT() to conclude traversal immediately on that custom-evaluated hit.

Performance Impact

By terminating the ray at the very first valid occluder, the GPU avoids dozens or hundreds of unnecessary ray-box and ray-triangle intersection tests. In scenes with heavy foliage, dense geometry, or multiple light sources, using terminateRayEXT() substantially reduces shader execution counts and memory bandwidth pressure across the ray tracing pipeline.