What Is rayPayloadEXT in GLSL Ray Tracing?

In modern GPU ray tracing using Vulkan and GLSL, the rayPayloadEXT storage qualifier defines a shared data structure that passes information between ray tracing shader stages. This article explains the primary function of rayPayloadEXT, how it facilitates communication between the ray generation, hit, and miss shaders, and how to use it efficiently in your rendering pipeline.

The Purpose of rayPayloadEXT

In the standard rasterization pipeline, shaders pass data down a fixed sequence of stages using standard in and out variables. Ray tracing, however, is non-linear and event-driven. A single ray can hit multiple surfaces, trigger recursive rays, or miss geometry entirely.

The rayPayloadEXT qualifier solves the problem of data transfer in this non-linear environment. It acts as a bidirectional communication block attached to an individual ray. When a shader initiates ray traversal, it associates a payload variable with that ray. The traversal system carries this payload across various stages—such as Any-Hit, Closest-Hit, and Miss shaders—allowing those stages to read context and write back results (such as accumulated radiance, surface normals, or hit distances).

How It Works Across Shader Stages

The payload lifecycle connects the calling shader to the executing hit or miss shaders:

Declaration and Invocation:
In the Ray Generation shader (or a Closest-Hit shader performing recursive ray tracing), you declare the payload using rayPayloadEXT and a layout location index:
```
layout(location = 0) rayPayloadEXT struct RayPayload {
    vec3 color;
    float distance;
    bool hit;
} prd; // Per-ray data
```
Passing to the Traversal Engine:
When invoking traceRayEXT(), the last parameter references the payload location index:
```
traceRayEXT(topLevelAS, rayFlags, cullMask, sbtRecordOffset, sbtRecordStride, missIndex, origin, tMin, direction, tMax, 0);
```
Here, 0 links the ray to the payload declared at layout(location = 0).
Receiving Data in Hit and Miss Shaders:
The receiving shaders (Closest-Hit, Any-Hit, or Miss) access this data by declaring the matching structure using the rayPayloadInEXT qualifier at the same location:
```
layout(location = 0) rayPayloadInEXT struct RayPayload {
    vec3 color;
    float distance;
    bool hit;
} prd;
```
These shaders read incoming parameters (like ray depth) and write their computational results directly into the payload fields. Once traversal finishes or the stage returns, control yields back to the caller, where the updated payload is read.

Performance Considerations

The memory backing a ray payload is allocated per active ray. To ensure optimal GPU utilization:

Minimize Payload Size: Large payloads consume register space and can spill into local GPU memory, reducing wave occupancy and overall performance. Pack variables into compact vector formats where possible.
Avoid Unused Fields: Only include data strictly necessary for the hit and miss evaluations of that specific ray type.
Use Distinct Payload Locations: Vulkan allows multiple payload locations. You can define a lightweight payload for shadow rays (e.g., a single boolean visibility flag) and a larger payload for primary camera rays, preventing simple visibility tests from incurring unnecessary memory overhead.