How traceRayEXT Uses cullMask in GLSL Ray Tracing

In Vulkan and GLSL ray tracing pipelines, the cullMask parameter in traceRayEXT acts as an 8-bit visibility filter that controls which Top-Level Acceleration Structure (TLAS) instances a ray can intersect. By evaluating a bitwise AND operation between the ray's cullMask and each instance's defined mask before traversing Bottom-Level Acceleration Structures (BLAS), the hardware can discard non-relevant geometry immediately. This article explains how cullMask functions, how it evaluates instance visibility, and how to effectively use it for performance and rendering control.

The Bitwise Masking Mechanism

During TLAS construction on the host side, each instance is assigned an 8-bit integer mask via the mask field of the VkAccelerationStructureInstanceKHR structure. When a ray is cast in GLSL using traceRayEXT, the third argument provides an 8-bit cullMask:

traceRayEXT(
    topLevelAS,           // acceleration structure
    rayFlags,             // ray flags (e.g., gl_RayFlagsOpaqueEXT)
    cullMask,             // 8-bit mask to filter instances
    sbtRecordOffset,      // SBT offset
    sbtRecordStride,      // SBT stride
    missIndex,            // miss shader index
    origin,               // ray origin
    tMin,                 // minimum hit distance
    direction,            // ray direction
    tMax,                 // maximum hit distance
    payloadLocation       // payload index
);

During traversal, the ray tracing hardware or driver evaluates the following condition for each candidate instance:

(ray.cullMask & instance.mask) != 0
  • If the result is non-zero: The ray considers the instance valid and proceeds to traverse the associated BLAS to test for primitive intersections.
  • If the result is zero: The instance is culled immediately. Traversal skips the entire BLAS associated with that instance without testing bounding boxes or triangles.

Only the least significant 8 bits of cullMask are evaluated; higher-order bits are ignored.

Common Use Cases

Using cullMask allows developers to define functional layers across the scene without maintaining multiple separate acceleration structures:

  • Shadow Ray Filtering: Instances can be flagged to determine whether they cast shadows. For example, assigning 0x01 to shadow-casting objects and tracing shadow rays with a cullMask of 0x01 ensures non-casting objects are never evaluated.
  • Reflection and Refraction Visibility: Virtual cameras or mirrors can restrict their ray tests to specific sets of objects by matching designated bit flags.
  • View-Dependent Layers: Gameplay layers, debugging volumes, or first-person view models can be assigned discrete bits (e.g., 0x04 for UI/first-person models) so primary camera rays hit them, while environment probes or global illumination rays exclude them by omitting that bit.

Performance Impact

Because the cullMask test occurs at the top level of the acceleration structure traversal, culled instances require zero memory reads or intersection calculations within their child nodes. Setting cullMask to 0xFF enables intersection against all instances with a non-zero instance mask, while setting it to 0x00 guarantees that the ray will miss all geometry and trigger the miss shader directly. Leveraging this bitwise rejection significantly reduces traversal overhead compared to checking material properties inside Any-Hit or Closest-Hit shaders.