Understanding rayQueryProceedEXT Traversal in GLSL

In Vulkan inline ray tracing via the GLSL_EXT_ray_query extension, rayQueryProceedEXT acts as an iterative driver for hardware-accelerated bounding volume hierarchy (BVH) traversal inside a shader loop. This function advances the ray through top-level and bottom-level acceleration structures (TLAS and BLAS), testing bounding boxes and primitives until it either encounters a candidate intersection that requires shader-side confirmation or exhausts all potential intersections. By returning a boolean value, it signals to the while loop whether traversal is actively paused at a candidate hit (true) or has completed entirely (false), allowing developers to query final intersection results.

Initialization and Ray State Setup

Before rayQueryProceedEXT can traverse geometry, the ray query object must be initialized using rayQueryInitializeEXT. This function sets:

  • The top-level acceleration structure (TLAS) handle.
  • Ray traversal flags (such as gl_RayFlagsOpaqueEXT or gl_RayFlagsTerminateOnFirstHitEXT).
  • The instance inclusion mask.
  • Geometric ray properties: world-space origin, minimum distance (\(t_{\min}\)), direction vector, and maximum distance (\(t_{\max}\)).

Initialization configures the internal traversal state machine and places the root node of the TLAS onto the hardware or emulation traversal stack.

How rayQueryProceedEXT Executes Traversal

Inside a while (rayQueryProceedEXT(rayQuery)) loop, the function operates as a stateful coroutine between the shader processor and the ray tracing pipeline (often fixed-function hardware such as RT Cores):

  1. Stack Processing and Node Testing: When called, the underlying implementation pops nodes from the BVH traversal stack. It performs ray-box intersection tests against internal nodes, descending through the TLAS into the corresponding BLAS instances based on transformation matrices and culling masks.
  2. Hardware-Level Hit Processing: If the ray hits fully opaque primitives and no custom shader handling is needed, the fixed-function unit automatically updates the internal "committed" hit distance (\(t\)) and continues traversal without yielding control back to the shader.
  3. Pausing Traversal for Candidate Intersections: The function halts traversal and returns true under conditions that require custom logic from the shader:
    • Non-Opaque Triangles: When encountering geometry flagged as non-opaque, traversal yields so the shader can evaluate alpha testing or procedural discard using rayQueryConfirmIntersectionEXT.
    • Axis-Aligned Bounding Boxes (AABBs): When intersecting custom procedural geometry defined by AABBs, traversal yields to let the shader perform an analytic intersection test and optionally submit a hit distance using rayQueryGenerateIntersectionEXT.
  4. Resuming Traversal: In the next iteration of the while loop, invoking rayQueryProceedEXT informs the traversal unit to resume from its current state, taking into account any confirmed or rejected hits.

Loop Termination

rayQueryProceedEXT returns false when traversal finishes, which immediately breaks the while loop. This occurs when:

  • The BVH traversal stack becomes empty, meaning all candidate paths have been tested or bounded out.
  • A flag such as gl_RayFlagsTerminateOnFirstHitEXT was set, and an acceptable intersection was confirmed.
  • The ray interval contracts to a point where remaining bounding boxes fall beyond the committed hit distance (\(t\)).

Post-Loop Resolution

Once the loop terminates, traversal is complete. The ray query object holds the final "committed" intersection data. The shader then uses functions such as rayQueryGetIntersectionTypeEXT(rayQuery, true) to check if a valid geometry hit was found, followed by retrieving primitive indices, instance IDs, barycentrics, and transformation matrices to compute shading or visibility.