How gl_RayFlagsSkipTrianglesEXT Restricts Ray Tests

In GLSL ray tracing via the GL_EXT_ray_tracing extension, gl_RayFlagsSkipTrianglesEXT modifies the traversal behavior of hardware ray tracing pipelines by bypassing all triangle-based geometry. This article explains how the flag operates within Vulkan acceleration structures, how it isolates ray-primitive intersection logic strictly to axis-aligned bounding boxes (AABBs), and how it coordinates with custom intersection shaders.

The Role of Acceleration Structures and Geometry Types

Bottom-Level Acceleration Structures (BLAS) can store two primary classes of geometry:

  1. Triangles: Built with vertex and index buffers. Ray intersections are evaluated via dedicated fixed-function ray-triangle intersection hardware.
  2. AABBs (Procedural Primitives): Defined by minimum and maximum spatial coordinates. Ray intersections test whether the ray intersects the bounding box volume, after which execution passes to an intersection shader.

Normally, the ray traversal unit traverses the bounding volume hierarchy (BVH) and tests both triangle primitives and AABBs encountered along the ray path.

Traversal Behavior with gl_RayFlagsSkipTrianglesEXT

When gl_RayFlagsSkipTrianglesEXT is passed into the rayFlags argument of traceRayEXT(), the hardware traversal engine alters its intersection culling logic:

  • Fixed-Function Triangle Culling: As the ray descends through the BVH, any leaf nodes marked as triangle geometry are discarded immediately. The fixed-function triangle test is never invoked, and triangle candidate hits are ignored.
  • Preservation of AABB Candidates: Leaf nodes containing custom AABB geometry remain fully active. When the ray intersects an AABB, traversal pauses to invoke the custom Intersection Shader (.rint) associated with that geometry index.
  • Shader Pipeline Targeting: Because triangle hits are eliminated at the traversal stage, triangle-specific any-hit shaders and closest-hit shaders are never triggered. Only hits reported by reportIntersectionEXT() inside the custom intersection shader will be considered valid hits.
// Example invocation restricting tests exclusively to bounding boxes
uint rayFlags = gl_RayFlagsSkipTrianglesEXT;

traceRayEXT(
    topLevelAS,             // Acceleration structure
    rayFlags,               // Traversal flags
    cullMask,               // Mask applied to instances
    sbtRecordOffset,        // Shader Binding Table offset
    sbtRecordStride,        // Shader Binding Table stride
    missIndex,              // Miss shader index
    origin,                 // Ray origin
    tMin,                   // Ray minimum distance
    direction,              // Ray direction
    tMax,                   // Ray maximum distance
    payloadLocation         // Payload index
);

Combined Flags and Interactions

gl_RayFlagsSkipTrianglesEXT is the inverse companion to gl_RayFlagsSkipAABBsEXT. Combining both flags will result in the ray missing all geometry entirely and directly invoking the miss shader.

Additionally, the flag operates downstream from instance masking:

  1. Instance Masking (cullMask): Evaluated first; whole instances can be discarded before primitive evaluation.
  2. Primitive Flag Filtering (gl_RayFlagsSkipTrianglesEXT): Discards triangle primitives within mixed-geometry BLASes or instance sets.
  3. Bounding Box Execution: Rays proceed solely to compute procedural intersections inside the bounding boxes.

Practical Applications

By enforcing ray tests strictly against AABBs without modifying or maintaining separate acceleration structures, gl_RayFlagsSkipTrianglesEXT is used for:

  • Volumetric and Signed Distance Field (SDF) Passes: Casting dedicated rays that only sample procedural mediums while ignoring the solid polygon meshes sharing the scene.
  • Particle and Point Cloud Queries: Intersecting custom mathematical primitives (such as oriented disks or bounding-box-contained metaballs) without collision against polygonal terrain.
  • Proxy and Trigger Collision Testing: Evaluating broad-phase or proxy triggers represented as AABBs in physics or visibility passes.