Understanding DXR Ray Gen, Closest Hit, and Miss Shaders

DirectX Raytracing (DXR) replaces the traditional rasterization pipeline with an event-driven shader model designed around ray traversal through bounding volume hierarchies. This article explains how the ray generation shader initiates rays, how hardware traversal determines intersections, and how control branches to either the closest hit shader or the miss shader to resolve lighting, materials, and environment data back to the final render target.

The Ray Generation Shader: Starting Traversal

The Ray Generation (RayGen) shader serves as the entry point for the DXR pipeline. Dispatched across a grid representing the render target dimensions, each thread corresponds to a specific pixel or sample.

The RayGen shader initializes a user-defined payload structure containing data such as radiance, hit distance, and recursion depth. It then calculates the world-space origin and direction of a ray based on the camera projection matrix and calls the intrinsic function TraceRay(). This invocation hands execution over to the fixed-function traversal hardware, targeting the Top-Level Acceleration Structure (TLAS).

Hardware Traversal and Resolution

Once TraceRay() is invoked, the GPU tests the ray against the acceleration structures to identify primitive intersections between the defined minimum (TMin) and maximum (TMax) distance bounds. Depending on whether a valid intersection is found, the pipeline branches execution to one of two stages:

  1. Geometry Intersected: If a valid hit occurs and passes any optional any-hit shader evaluation, the pipeline identifies the intersection closest to the ray origin and invokes the Closest Hit Shader.
  2. No Geometry Intersected: If the ray exits the scene bounds without striking any geometry, the pipeline invokes the Miss Shader.

The Closest Hit Shader: Surface Evaluation

The Closest Hit (CHS) shader executes when a ray successfully strikes opaque or evaluated geometry. It has access to the ray payload, built-in system values (such as the primitive ID and ray direction), and intersection attributes (such as barycentric coordinates).

The CHS typically performs surface shading by:

  • Interpolating vertex attributes across the hit triangle to sample diffuse, normal, and roughness textures.
  • Evaluating local lighting models using analytic light sources.
  • Writing the calculated surface color or material properties into the ray payload.

The CHS can also invoke TraceRay() to cast secondary rays, such as shadow rays toward light sources or reflection/refraction rays. In these cases, the current CHS acts similarly to a RayGen shader, waiting for the secondary ray traversal to complete before using the updated secondary payload to resolve the final surface appearance.

The Miss Shader: Environment Handling

The Miss Shader (MS) executes when a ray traverses to TMax without registering an intersection. Because no geometry was encountered, the shader cannot sample mesh attributes. Instead, it typically resolves background information.

Common uses for the Miss Shader include:

  • Sampling high-dynamic-range (HDR) environment maps or procedural sky models using the ray direction.
  • Returning a default ambient light value.
  • Setting a boolean flag in the payload for shadow rays, indicating that the path to a light source is unobstructed.

Like the CHS, the Miss Shader writes its results directly into the ray payload before terminating.

Payload Return and Output

After either the Closest Hit Shader or the Miss Shader completes execution, control returns to the shader that called TraceRay(). If TraceRay() was called recursively within a CHS, control returns to that CHS. Once all secondary rays resolve, control returns to the originating RayGen shader.

The RayGen shader unpacks the final values stored in the payload, performs any necessary post-processing or temporal accumulation, and writes the output pixel color directly to an unordered access view (UAV) bound to the render target.