How rayPayloadInEXT Works in GLSL Ray Tracing

In GLSL ray tracing pipelines using the GL_EXT_ray_tracing extension, rayPayloadInEXT is a storage qualifier used in downstream execution stages—such as Closest Hit, Any Hit, and Miss shaders—to access and modify the data payload passed from an invoking shader. It establishes an inter-stage communication channel that pairs directly with a caller's rayPayloadEXT variable through explicit layout location indices provided during a traceRayEXT call. This mechanism enables shaders to read incoming ray states, perform computations, and write results back to the caller across ray traversal boundaries.

The Location Matching Mechanism

Communication between the caller (typically a Ray Generation shader or a recursive Closest Hit shader) and the callee (Closest Hit, Any Hit, or Miss shaders) relies on explicit layout location indices.

The caller declares a payload variable using the rayPayloadEXT qualifier:

layout(location = 0) rayPayloadEXT vec3 hitColor;

When invoking ray traversal via traceRayEXT, the caller specifies the target payload location index as an argument:

traceRayEXT(topLevelAS, rayFlags, cullMask, sbtRecordOffset, sbtRecordStride, missIndex, origin, tMin, direction, tMax, 0);

The final parameter (0) binds execution to payload location 0. The downstream shader stage declares its receiving end using rayPayloadInEXT at the identical location:

layout(location = 0) rayPayloadInEXT vec3 hitColor;

The driver and hardware use this matched location slot to map the memory backing the payload structure across the pipeline stages.

Read and Write Semantics

Unlike standard vertex-to-fragment input variables in rasterization pipelines (which are strictly read-only), rayPayloadInEXT acts as a mutable reference to the storage defined by the caller.

  1. Initialization: The caller writes initial data to its rayPayloadEXT variable (e.g., ray depth, throughput, or default visibility values) before calling traceRayEXT.
  2. Transfer: When an intersection or miss occurs, the execution context transitions to the designated shader stage. The runtime points the callee's rayPayloadInEXT variable directly to that memory space.
  3. Modification: The downstream shader reads incoming data from rayPayloadInEXT to make shading decisions and writes output data directly back into rayPayloadInEXT (e.g., surface radiance, normal vectors, or hit distances).
  4. Return: Once the downstream shader finishes execution, control returns to the caller immediately following the traceRayEXT instruction. The caller's rayPayloadEXT variable now reflects all modifications made by the callee.

Supported Data Types and Memory Footprint

The payload variable can be an intrinsic type (such as float or vec4) or a user-defined struct. Because ray payloads represent persistent state during traversal and recursive calls, they consume hardware register or stack space. Keeping rayPayloadInEXT structures compact and aligned to 4-byte boundaries is critical for minimizing memory traffic and maximizing occupancy across ray tracing pipelines.