Optimizing DXR Traversal with Ray Flags

Ray flag management in DirectX Raytracing (DXR) allows developers to optimize the traversal pipeline by pruning unnecessary shader dispatches, reducing register pressure, and staying within fixed-function hardware paths. By passing specific flags during TraceRay calls, the runtime and underlying GPU hardware bypass costly programmable stages, such as the Closest-Hit and Any-Hit shaders, significantly accelerating visibility tests, shadow calculations, and scene queries.

The Role of RAY_FLAG_SKIP_CLOSEST_HIT_SHADER

The RAY_FLAG_SKIP_CLOSEST_HIT_SHADER flag instructs the traversal engine to omit executing the Closest-Hit Shader (CHS) when an intersection occurs. Under normal traversal conditions, the ray tracing pipeline tracks the closest intersection along the ray interval and dispatches the associated CHS once traversal concludes.

Executing a CHS introduces overhead:

  • Shader Invocation Overhead: The GPU must schedule a programmable shader stage, switch execution contexts, and manage register allocation.
  • Payload Serialization: CHS execution typically involves writing hit attributes, material parameters, and lighting contributions back to the ray payload, increasing memory traffic.

When computing boolean visibility queries, such as shadow rays or ambient occlusion occlusion tests, the exact surface properties at the hit location are irrelevant. Skipping the CHS entirely removes this execution step, allowing the ray state to resolve immediately upon completing traversal.

RAY_FLAG_SKIP_CLOSEST_HIT_SHADER is most potent when combined with RAY_FLAG_ACCEPT_FIRST_HIT_AND_END_SEARCH. By default, DXR traverses the entire Bounding Volume Hierarchy (BVH) to locate the t parameter corresponding to the nearest primitive.

When both flags are used together:

  1. The hardware terminates traversal immediately upon detecting any valid intersection.
  2. The CHS dispatch is suppressed.
  3. The ray payload defaults to an occluded state, or a Miss Shader is used to flag visibility if no intersection is found.

This combination reduces a full BVH search into a short-circuit test, minimizing memory fetches across BVH internal nodes and primitive leaf nodes.

Bypassing Any-Hit Shaders with RAY_FLAG_FORCE_OPAQUE

Traversal frequently stalls when entering programmable Any-Hit Shaders (AHS), which are often used for alpha testing. Each AHS execution halts fixed-function ray traversal to run programmable code on the GPU's Compute Units/Streaming Multiprocessors.

Setting RAY_FLAG_FORCE_OPAQUE treats all geometry as opaque regardless of how the bottom-level acceleration structure (BLAS) geometry flags are configured. This guarantees that:

  • No Any-Hit Shaders will be invoked during BVH traversal.
  • Dedicated ray tracing hardware cores (such as RT Cores) can evaluate box and triangle intersections entirely within fixed-function silicon without context-switching to programmable compute.

Culling Flags and Triangle Intersection Workloads

Flags such as RAY_FLAG_CULL_BACK_FACING_TRIANGLES, RAY_FLAG_CULL_FRONT_FACING_TRIANGLES, and RAY_FLAG_CULL_NON_OPAQUE prune primitives before the intersection test is calculated. Ray-triangle intersection tests consume silicon resources and bandwidth. Discarding primitives by winding order or opacity state at the hardware traversal level prevents unnecessary hit evaluations and associated payload updates.

Summary of Performance Impacts

Proper ray flag configuration optimizes DXR traversal through several hardware mechanisms:

  • Eliminating Programmable Stalls: Bypassing CHS and AHS execution keeps traversal within dedicated fixed-function hardware.
  • Reducing Ray Payload Footprint: When fewer shaders modify data, payloads require fewer registers, maximizing GPU occupancy and ray parallelism.
  • Early Traversal Termination: Combining skip-hit flags with early-out flags reduces BVH node fetches, significantly lowering cache misses and VRAM bandwidth consumption.