Ray Generation Shaders in GLSL GL_EXT_ray_tracing

Ray generation shaders serve as the initial entry point for hardware-accelerated ray tracing pipelines using the GL_EXT_ray_tracing GLSL extension. This article breaks down the foundational structure required to write a ray generation shader—including necessary extension directives, resource bindings, payload setup, and built-in variables—and explains how the shader calculates world-space rays, triggers traversal via traceRayEXT, and executes across a dispatched computational grid.

Core Structure and Required Declarations

A ray generation shader must target SPIR-V and explicitly enable the ray tracing extension. The top of the shader file requires #version 460 and the extension directive:

#version 460
#extension GL_EXT_ray_tracing : require

Essential Bindings

Ray generation shaders typically bind three primary resources:

  1. Top-Level Acceleration Structure (TLAS): Declared using the accelerationStructureEXT uniform type to allow traversal over scene geometry.
  2. Storage Image: A 2D image (e.g., rgba8 or rgba32f) where final pixel color values are written.
  3. Uniform or Storage Buffers: Camera matrices (inverse view and inverse projection) used to compute ray vectors from screen-space coordinates.

Ray Payloads

Data shared between the ray generation shader, hit shaders, and miss shaders is stored in a ray payload. The payload is declared in global scope using the rayPayloadEXT storage qualifier and an explicit location index:

layout(location = 0) rayPayloadEXT vec3 hitValue;

Built-In Variables

The execution environment provides several built-in variables that allow each invocation to determine its position within the launch grid:

  • gl_LaunchIDEXT: A uvec3 indicating the coordinates of the current thread invocation. Typically, gl_LaunchIDEXT.xy corresponds to the pixel coordinate \((x, y)\).
  • gl_LaunchSizeEXT: A uvec3 representing the total dimensions of the ray tracing dispatch (e.g., width, height, and depth).

Ray Calculation and traceRayEXT

Each invocation derives normalized device coordinates (NDC) from gl_LaunchIDEXT.xy relative to gl_LaunchSizeEXT.xy. Transforming these coordinates by the inverse view and projection matrices yields the ray's origin and direction in world space.

Traversing the acceleration structure is initiated using the built-in function traceRayEXT. The signature is structured as follows:

void traceRayEXT(
    accelerationStructureEXT topLevel,
    uint rayFlags,
    uint cullMask,
    uint sbtRecordOffset,
    uint sbtRecordStride,
    uint missIndex,
    vec3 origin,
    float tMin,
    vec3 direction,
    float tMax,
    int payload
);
  • topLevel: The bound TLAS.
  • rayFlags: Traversal control flags (e.g., gl_RayFlagsOpaqueEXT or gl_RayFlagsNoneEXT).
  • cullMask: An 8-bit mask evaluated against instance masks to filter geometry.
  • sbtRecordOffset: Offset into the Shader Binding Table (SBT) for hit groups.
  • sbtRecordStride: Stride between hit groups in the SBT.
  • missIndex: The index of the miss shader to invoke if no geometry is intersected.
  • origin / direction: World-space origin point and normalized direction vector.
  • tMin / tMax: Geometric clipping distances along the ray.
  • payload: The payload layout location number matching the rayPayloadEXT variable.

Complete Minimal Shader Example

#version 460
#extension GL_EXT_ray_tracing : require

layout(binding = 0, set = 0) uniform accelerationStructureEXT topLevelAS;
layout(binding = 1, set = 0, rgba8) uniform image2D outputImage;
layout(binding = 2, set = 0) uniform CameraData {
    mat4 invView;
    mat4 invProj;
} camera;

layout(location = 0) rayPayloadEXT vec3 prd;

void main() {
    const vec2 pixelCenter = vec2(gl_LaunchIDEXT.xy) + vec2(0.5);
    const vec2 inUV = pixelCenter / vec2(gl_LaunchSizeEXT.xy);
    vec2 d = inUV * 2.0 - 1.0;

    vec4 origin = camera.invView * vec4(0.0, 0.0, 0.0, 1.0);
    vec4 target = camera.invProj * vec4(d.x, d.y, 1.0, 1.0);
    vec4 direction = camera.invView * vec4(normalize(target.xyz), 0.0);

    uint rayFlags = gl_RayFlagsOpaqueEXT;
    float tMin = 0.001;
    float tMax = 10000.0;

    traceRayEXT(
        topLevelAS,
        rayFlags,
        0xFF,
        0,
        0,
        0,
        origin.xyz,
        tMin,
        direction.xyz,
        tMax,
        0
    );

    imageStore(outputImage, ivec2(gl_LaunchIDEXT.xy), vec4(prd, 1.0));
}

Execution Flow

  1. Pipeline Invocation: The host API issues a trace command (such as vkCmdTraceRaysKHR), defining the 3D dimensions of the dispatch grid.
  2. Independent Spawning: The GPU spawns individual threads corresponding to gl_LaunchSizeEXT. Unlike compute shaders, there is no shared memory or explicit threadgroup/workgroup synchronization.
  3. Ray Traversal: When traceRayEXT executes, execution within the thread yields while fixed-function hardware traverses the bounding volume hierarchy (BVH).
  4. Hit/Miss Execution: Intersection, any-hit, closest-hit, or miss shaders execute according to traversal results, updating the memory referenced by the ray payload location.
  5. Resumption: Once traversal and subsequent shader executions finish, traceRayEXT returns control back to the ray generation shader.
  6. Output Writing: The shader reads the updated payload values, performs any required post-processing, and commits the result to the destination resource via operations like imageStore.