DirectX Raytracing Payload Limit and Performance

This article examines the DirectX Raytracing (DXR) ray payload limit, explaining what the payload is, how the API handles data passing between shaders, and the direct impact of payload sizing on GPU register allocation, memory bandwidth, and overall shader performance.

What Is the DXR Ray Payload?

In DirectX Raytracing, a ray payload is a user-defined data structure passed as an inout parameter to TraceRay(). It serves as the primary communication bridge between the initiating shader (such as a Ray Generation shader) and the downstream shaders that evaluate the ray's trajectory (Closest Hit, Any Hit, and Miss shaders). Shaders write data such as hit distances, surface normals, material identifiers, and accumulated color directly into this structure.

In DXR 1.0, the pipeline configuration specifies the maximum payload size across all shaders via the RaytracingPipelineConfig subobject. DXR 1.1 expanded this with RaytracingPipelineConfig1, allowing developers to set exact byte limits to help the compiler optimize memory allocation.

API Limits vs. Hardware Constraints

The DirectX specification establishes a formal upper bound on payload size—typically up to 4,096 bytes (4 KB). However, this architectural ceiling is theoretical. Writing a shader that uses anywhere near 4 KB of payload data will cause severe performance degradation on modern GPUs.

Modern graphics hardware does not treat the payload as a static buffer in fast cache; instead, the compiler attempts to allocate the payload directly into GPU Vector General-Purpose Registers (VGPRs). When the payload size exceeds the available physical registers, the driver is forced to store the excess data in high-latency scratch memory (VRAM).

How Payload Size Influences Shader Performance

The size of the ray payload directly dictates several critical performance characteristics:

1. Register Pressure and Occupancy

GPUs hide latency through massive parallelism by executing thousands of threads simultaneously organized into warps or wavefronts. Every byte added to a ray payload consumes additional registers. As register usage per thread increases, the GPU scheduler can maintain fewer active wavefronts concurrently on a Streaming Multiprocessor (SM) or Compute Unit (CU). This drop in occupancy diminishes the hardware’s ability to hide memory latency during texture sampling and acceleration structure traversal.

2. Register Spilling

If a ray payload becomes too large to fit within the register budget allocated for a shader stage, the compiler triggers register spilling. Spilled data is moved to local off-chip memory or private scratch buffers. Because ray traversal is already memory-bandwidth intensive, forcing the hardware to constantly read and write spilled payload states across hit stages creates severe memory pipeline stalls.

3. Stack Overhead in Recursive Ray Tracing

When shaders call TraceRay() recursively (for example, shooting reflection rays from within a Closest Hit shader), the current payload state must be preserved across ray calls. The GPU maintains a ray-tracing stack where it pushes the payload and local state before processing the child ray. A larger payload drastically scales the memory footprint of this stack by:

\[\text{Total Stack Memory} = \text{Payload Size} \times \text{Recursion Depth} \times \text{Active Threads}\]

This rapidly exhausts L1/L2 caches and forces expensive VRAM round-trips.

4. Ray Divergence and Serialization

Ray payloads frequently introduce divergent code paths. When different rays in the same warp take distinct branches based on complex payload fields, execution must be serialized. Larger payloads also require more ALU instructions simply to pack, unpack, and initialize fields across different hit programs.

Optimization Strategies for Payloads

To minimize payload overhead and sustain high frame rates:

  • Split Payloads by Ray Type: Never use a single monolithic payload for every ray. A shadow ray only requires a single boolean or 1-bit flag (IsHit), while a primary radiance ray requires material and color attributes. Define separate structures and register distinct hit groups for each.
  • Pack Data Types: Quantize floating-point values wherever possible. Surface normals can often be packed into 32-bit octahedral representations or 16-bit half-floats instead of three 32-bit floats. Pack multiple boolean flags into a single bitfield.
  • Leverage Ray Queries: When full shader re-binding is unnecessary (such as for ambient occlusion or simple shadows), utilize DXR 1.1 Inline Raytracing (RayQuery). This executes traversal entirely within the calling shader, completely bypassing payload setup and stack overhead.