How traceRayEXT Launches Secondary Rays in GLSL

In GLSL ray tracing pipelines using the GL_EXT_ray_tracing extension, traceRayEXT is the core intrinsic used to initiate ray traversal through a scene hierarchy. While typically called first within a ray generation shader to cast primary camera rays, traceRayEXT can also be invoked from within hit shaders (such as closest hit shaders) to cast secondary rays for effects like reflections, refractions, ambient occlusion, and shadows. This article explains the technical mechanics of traceRayEXT, the parameters required to define a secondary ray, and how the underlying hardware processes secondary traversal through top-level acceleration structures.

The Role of Acceleration Structures

Secondary rays do not intersect raw geometry directly; instead, they query a two-level acceleration structure (TLAS and BLAS). The Top-Level Acceleration Structure (TLAS) holds instances of objects, transforms, and visibility masks, while the Bottom-Level Acceleration Structures (BLAS) contain the actual geometric primitives (triangles or AABBs).

When casting a secondary ray, the shader passes the uniform accelerationStructureEXT handle representing the TLAS. The traversal engine queries this structure to locate the nearest or any primitive intersecting the secondary ray’s path.

Function Signature and Parameters

The traceRayEXT function is declared as follows:

void traceRayEXT(
    accelerationStructureEXT topLevelAS,
    uint rayFlags,
    uint cullMask,
    uint sbtRecordOffset,
    uint sbtRecordStride,
    uint missIndex,
    vec3 origin,
    float Tmin,
    vec3 direction,
    float Tmax,
    int payloadLocation
);
  • topLevelAS: The scene's acceleration structure to traverse.
  • rayFlags: Traversal behavior modifiers, such as gl_RayFlagsOpaqueEXT (to skip any-hit shaders) or gl_RayFlagsTerminateOnFirstHitEXT (ideal for shadow/visibility rays).
  • cullMask: An 8-bit mask tested against each instance’s mask in the TLAS to conditionally skip non-relevant geometry.
  • sbtRecordOffset and sbtRecordStride: Values used to calculate hit shader indices within the Shader Binding Table (SBT), determining which intersection, any-hit, or closest-hit shaders are executed.
  • missIndex: The index offset in the SBT indicating which miss shader runs if the secondary ray misses all geometry.
  • origin and direction: The world-space coordinates defining the secondary ray.
  • Tmin and Tmax: The parametric interval limiting the ray segment. Setting an appropriate Tmin (e.g., 0.001) helps avoid surface acne from self-intersection.
  • payloadLocation: The layout location index of the ray payload structure where results from the secondary ray are written.

Launching Secondary Rays from Hit Shaders

To launch a secondary ray, the shader must declare a designated payload storage qualifier matching the payloadLocation parameter:

layout(location = 1) rayPayloadEXT RayPayload secondaryPayload;

Within a closest-hit shader, secondary rays are derived from the hit attributes of the primary ray:

  1. Calculate Ray Geometry: The world-space intersection point is calculated using the primary ray parameters (gl_WorldRayOriginEXT + gl_WorldRayDirectionEXT * gl_HitTEXT). The new ray direction is computed based on lighting vectors (for shadows) or surface normal reflection (for mirrors).
  2. Offset Origin: To prevent the ray from re-intersecting the originating surface due to floating-point imprecision, the origin is nudged slightly along the surface normal or Tmin is set to a small non-zero value.
  3. Invoke traceRayEXT: The function is executed, passing the TLAS handle and the secondary payload location.

Traversal and Execution Flow

When traceRayEXT executes for a secondary ray:

  1. Pipeline State Pause: Execution of the calling shader is paused at the call site.
  2. Hardware Traversal: The ray traversal unit intersects bounding boxes and triangles in the TLAS and corresponding BLAS instances according to cullMask and rayFlags.
  3. Shader Invocation: Depending on ray path interactions, the pipeline executes relevant intersection or any-hit shaders. Upon finding the definitive hit, the designated closest hit shader runs, modifying secondaryPayload. If no hit occurs within [Tmin, Tmax], the specified miss shader executes.
  4. Resumption: Control returns to the calling hit shader directly after the traceRayEXT instruction, where it reads secondaryPayload to composite the final surface color.

Pipeline Recursion Limits

To use traceRayEXT recursively (e.g., secondary rays calling tertiary rays), the pipeline must be configured via the Vulkan API with a maxPipelineRayRecursionDepth greater than 1. Because deep recursion places substantial pressure on execution stack memory, multi-bounce secondary rays are often unrolled iteratively in the ray generation shader using an explicit loop and state payload instead of relying on recursive hit shader calls.