How gl_RayFlagsCullNoOpaqueEXT Works in GLSL

In GLSL ray tracing, gl_RayFlagsCullNoOpaqueEXT is a traversal control flag passed to traceRayEXT that instructs the ray tracing hardware or driver to automatically skip any geometry identified as non-opaque. By discarding non-opaque primitives early during bounding volume hierarchy (BVH) traversal, the pipeline avoids executing expensive any-hit shaders and prevents these surfaces from registering intersections. This article explains how the ray traversal engine determines opacity, how the cull flag suppresses non-opaque candidates, and when to use it for optimal performance.

Determining Primitive Opacity

Before gl_RayFlagsCullNoOpaqueEXT can cull a primitive, the traversal engine must evaluate the effective opacity of the candidate geometry. Opacity is determined through a hierarchy of acceleration structure flags, instance flags, and ray flags:

  1. Geometry Flags: During acceleration structure building, geometries can be marked with VkGeometryFlagBitsKHR such as VK_GEOMETRY_OPAQUE_BIT_KHR.
  2. Instance Flags: Acceleration structure instances can override geometry states using VK_GEOMETRY_INSTANCE_FORCE_OPAQUE_BIT_KHR or VK_GEOMETRY_INSTANCE_FORCE_NO_OPAQUE_BIT_KHR.
  3. Ray Flags: Flags supplied to traceRayEXT, such as gl_RayFlagsOpaqueEXT or gl_RayFlagsNoOpaqueEXT, can globally force the opacity state of all encountered geometry along that ray.

A primitive is classified as non-opaque if it lacks an opaque flag and has not been forced to opaque by an instance or ray flag. By default, non-opaque primitives require an any-hit shader invocation to evaluate whether an intersection is accepted (for example, evaluating alpha masks or procedural cutouts).

The Culling Mechanism During Traversal

When a ray is cast with gl_RayFlagsCullNoOpaqueEXT, the traversal engine modifies how candidate primitive intersections are handled:

  • Intersection Testing: The ray continues its standard traversal through the BVH nodes. When an intersection candidate is found, the engine reads the resolved opacity state of that geometry.
  • Early Discard: If the geometry is resolved as non-opaque, the traversal logic immediately rejects the intersection. The candidate is dropped without recording a hit.
  • Any-Hit Shader Bypass: Because the primitive is culled immediately at the traversal stage, the pipeline never schedules or invokes the associated any-hit shader.
  • Closest-Hit Filtering: Since culled primitives never register as valid hits, they cannot shorten gl_RayTmaxEXT or become the final hit recorded for the closest-hit shader.

Only primitives strictly resolved as opaque are allowed to generate valid hits and update ray intersection state.

Practical Implications and Use Cases

The primary benefit of gl_RayFlagsCullNoOpaqueEXT is performance. Any-hit shaders interrupt hardware-accelerated traversal routines, creating thread divergence and memory stalls. By discarding non-opaque candidates directly in the traversal loop, hardware pipelines execute with minimal interruption.

This flag is commonly used in scenarios such as:

  • Coarse Shadow Rays: Tracing shadow rays where partial occluders (e.g., foliage or glass) are either ignored entirely or handled in a separate pass.
  • Ambient Occlusion: Calculating ambient occlusion against solid surfaces only, avoiding the computational overhead of testing transparency maps.
  • Multi-Pass Ray Routing: Splitting scene evaluations into distinct opaque and transparent passes to optimize divergence and memory access patterns.