How to Use rayQueryInitializeEXT in GLSL Shaders

This article provides an overview of how the rayQueryInitializeEXT function sets up inline ray casting within a standard GLSL compute shader. It explains the purpose of the Ray Query extension (GL_EXT_ray_query), the specific parameters required to configure a ray query object, and how the function bridges traditional compute workflows with hardware-accelerated ray tracing hardware without requiring dedicated ray tracing pipelines.

Inline ray casting via ray queries allows traditional compute, fragment, or vertex shaders to trace rays against a Bounding Volume Hierarchy (BVH) without transitioning to a dedicated ray tracing pipeline (such as Ray Generation, Closest Hit, and Miss shaders). To enable this in a compute shader, you must include #extension GL_EXT_ray_query : require and declare an acceleration structure uniform:

layout(binding = 0, set = 0) uniform accelerationStructureEXT topLevelAS;

Within your main() compute function, an uninitialized query object is declared using the rayQueryEXT type:

rayQueryEXT rayQuery;

The rayQueryInitializeEXT function prepares this object for traversal. It binds the query object to a specific acceleration structure, specifies traversal behavior, sets visibility masking, and defines the geometric properties of the ray in world space.

The signature of the function is:

void rayQueryInitializeEXT(
    rayQueryEXT rayQuery,
    accelerationStructureEXT topLevelAS,
    uint rayFlags,
    uint cullMask,
    vec3 origin,
    float tMin,
    vec3 direction,
    float tMax
);

Parameter Breakdown

  1. rayQuery: The in-out variable storing the traversal state machine on the shader thread.
  2. topLevelAS: The Top-Level Acceleration Structure (TLAS) containing the scene geometry instances to test against.
  3. rayFlags: A bitmask controlling hardware traversal behavior. Common flags include gl_RayFlagsOpaqueEXT (treats all geometry as opaque, bypassing any-hit shaders), gl_RayFlagsTerminateOnFirstHitEXT (stops traversal at the first hit, ideal for shadow rays), and gl_RayFlagsNoneEXT.
  4. cullMask: An 8-bit integer mask evaluated against the instance mask in the TLAS. An instance is tested only if (cullMask & instance.mask) != 0. Setting this to 0xFF tests against all instances.
  5. origin: A vec3 representing the starting point of the ray in world space.
  6. tMin: A float setting the minimum distance along the ray vector before intersections are considered valid. This prevents self-intersection artifacts (acne).
  7. direction: A vec3 representing the ray direction vector. It does not strictly need to be normalized, but normalizing simplifies distance calculations.
  8. tMax: A float setting the maximum travel distance of the ray.

Traversal and Consumption

rayQueryInitializeEXT only configures the internal state of the ray query; it does not perform the intersection testing itself. Traversal occurs immediately after initialization through a loop controlled by rayQueryProceedEXT:

rayQueryInitializeEXT(rayQuery, topLevelAS, gl_RayFlagsTerminateOnFirstHitEXT | gl_RayFlagsOpaqueEXT, 0xFF, rayOrigin, 0.001, rayDirection, 1000.0);

while (rayQueryProceedEXT(rayQuery)) {
    // Process candidate intersections for non-opaque geometry if necessary
}

if (rayQueryGetIntersectionTypeEXT(rayQuery, true) != gl_RayQueryCommittedIntersectionNoneEXT) {
    // A hit was confirmed
}

By passing these parameters, rayQueryInitializeEXT sets up registers within the GPU ray tracing units directly from the compute thread, enabling flexible tasks such as ambient occlusion, shadow evaluation, and indirect visibility checks within a standard compute dispatch.