gl_RayFlagsSkipAABBsEXT in GLSL Ray Queries

This article explores the function, mechanics, and performance implications of the gl_RayFlagsSkipAABBsEXT flag in Vulkan ray queries using GLSL. In hardware-accelerated ray tracing, scenes often mix standard triangle geometry with custom procedural primitives bound by Axis-Aligned Bounding Boxes (AABBs). Setting gl_RayFlagsSkipAABBsEXT instructs the traversal hardware to ignore all AABB primitives, streamlining ray intersection tests, eliminating shader overhead, and boosting performance when procedural geometry is irrelevant to a specific rendering pass.

Ray Queries and Procedural Geometry

In Vulkan's VK_KHR_ray_query (and the corresponding GLSL extension GL_EXT_ray_query), ray traversal is executed inline within existing shader stages like compute, fragment, or mesh shaders. Acceleration structures house two primary types of geometry:

  1. Triangles: Intersected automatically by fixed-function hardware.
  2. AABBs (Procedural Primitives): User-defined bounding boxes defining custom geometries, such as hair, curves, metaballs, or volumetric shapes.

When a ray query encounters an AABB, hardware traversal cannot evaluate the final hit directly because it does not understand the internal shape. Instead, the ray query halts and reports a candidate intersection of type gl_RayQueryCandidateIntersectionAABBEXT. The shader must then execute custom intersection math and explicitly confirm or ignore the hit using rayQueryConfirmIntersectionEXT or rayQueryGenerateIntersectionEXT.

The Mechanism of gl_RayFlagsSkipAABBsEXT

The gl_RayFlagsSkipAABBsEXT flag is passed as a ray flag parameter to rayQueryInitializeEXT(). When applied, it directs the ray traversal engine to bypass all AABB bounding primitives encountered in the bottom-level acceleration structures (BLAS).

Under the hood, the hardware's traversal unit checks the primitive type during bounding box hierarchy (BVH) node parsing. If the primitive or node is flagged as procedural/AABB geometry and gl_RayFlagsSkipAABBsEXT is active:

  • The traversal unit will not generate a candidate hit for the AABB.
  • The ray query will not trigger an AABB intersection loop in the shader code.
  • Traversal continues uninterrupted, evaluating only triangle primitives.

Practical Applications

Using this flag is particularly useful in several rendering workflows:

  • Selective Shadow Rays: Certain procedural geometries, such as volumetric bounding spheres, particle effects, or UI overlays, may not need to cast hard shadows. Discarding them via ray flags prevents costly shader-side intersection evaluations.
  • Ambient Occlusion (AO): High-frequency procedural shapes like complex curve hair may cause noise or excessive evaluation time during broad AO calculations where fine detail is unneeded.
  • Separation of Geometry Passes: In systems using hybrid pipelines, passes calculating visibility specifically against static, opaque polygonal environments can completely strip out overhead caused by adjacent dynamic procedural content.

Performance Impact

Procedural primitive handling during ray queries is typically expensive because it interrupts traversal, transferring execution back to the shader code. In SIMD/SIMT architectures, if a single thread in a warp or subgroup hits an AABB, the entire warp may suffer from divergence while resolving the candidate intersection.

By specifying gl_RayFlagsSkipAABBsEXT, developers eliminate traversal interruption and warp divergence related to procedural primitives. The ray traversal engine remains entirely on fixed-function hardware paths until it finishes or intersects relevant triangle primitives, minimizing register pressure and maximizing execution throughput.