gl_RayFlagsNoOpaqueEXT Effect on Any-Hit Shaders

In GLSL ray tracing, gl_RayFlagsNoOpaqueEXT fundamentally alters how ray traversal handles geometry opacity, directly impacting whether any-hit shaders are executed. This article provides a technical breakdown of how setting this flag overrides default geometry properties within acceleration structures, forces the invocation of any-hit shaders for candidate ray-primitive intersections, and affects the overall ray traversal pipeline.

Operational Mechanism of gl_RayFlagsNoOpaqueEXT

In the standard ray tracing pipeline, an intersection candidate’s opacity determines whether traversal triggers an any-hit shader or skips directly to accepting the hit. Geometry can be defined as opaque either through acceleration structure build flags (such as VK_GEOMETRY_OPAQUE_BIT_KHR) or instance flags. When geometry is deemed opaque, hardware traversal automatically skips any-hit shader execution to optimize performance.

Setting the gl_RayFlagsNoOpaqueEXT flag when launching a ray (via traceRayEXT) overrides these acceleration structure settings. It forces the traversal engine to treat all encountered geometry along the ray's path as non-opaque, regardless of how the bottom-level acceleration structure (BLAS) or top-level acceleration structure (TLAS) originally configured those primitives.

Direct Impact on Any-Hit Shaders

When gl_RayFlagsNoOpaqueEXT is active, the operational effects on the any-hit shader are:

  1. Forced Invocation: Traversal will execute the bound any-hit shader for every potential intersection candidate along the ray. Primitives that would normally bypass the shader due to opaque flags will now execute it.
  2. Hit Evaluation Delegation: The determination of whether a hit is accepted or ignored is handed back to shader code. The any-hit shader can programmatically accept the hit or call ignoreIntersectionEXT() to discard it (commonly used for alpha testing or stenciled cutouts).
  3. Requirement of a Bound Shader: If an intersection occurs and the assigned hit group contains an any-hit shader, it will be invoked. If the hit group does not define an any-hit shader, the primitive is still treated as non-opaque, but traversal simply accepts the intersection without executing user code.

Pipeline Considerations and Precedence

  • Conflict with gl_RayFlagsOpaqueEXT: Ray flags are mutually exclusive in their intent. Specifying both gl_RayFlagsNoOpaqueEXT and gl_RayFlagsOpaqueEXT on the same ray launch produces undefined behavior.
  • Instance Flag Precedence: gl_RayFlagsNoOpaqueEXT takes precedence over geometry-level opaque declarations, ensuring that no primitive encountered by this specific ray can bypass transparency evaluation.
  • Performance Overhead: Forcing non-opaque evaluation disables hardware-level optimizations that fast-path ray termination on the first hit or bypass shader invocations. Consequently, invoking any-hit shaders on all geometry increases shader execution frequency and can substantially impact traversal throughput.