Understanding gl_InstanceCustomIndexEXT in GLSL Ray Tracing

This article provides an overview of the gl_InstanceCustomIndexEXT built-in variable used in GLSL ray tracing pipelines. It covers what the variable represents, where its value originates from the host API, which shader stages can access it, and how graphics developers use it to efficiently route per-instance data such as materials and resource descriptors.

What is gl_InstanceCustomIndexEXT?

In GLSL ray tracing pipelines (enabled via the GL_EXT_ray_tracing extension), gl_InstanceCustomIndexEXT is a built-in input variable of type int. It exposes a user-defined integer value assigned to a specific geometry instance in the Top-Level Acceleration Structure (TLAS).

When a ray intersects a primitive, the pipeline determines which instance was struck and makes this custom integer available to the relevant ray tracing shaders.

Origin in the Vulkan API

The value of gl_InstanceCustomIndexEXT does not originate inside the shader; it is defined on the CPU when setting up the acceleration structures.

In the Vulkan API, a TLAS is composed of instances defined by the VkAccelerationStructureInstanceKHR structure. This structure contains a 24-bit bitfield member named instanceCustomIndex:

typedef struct VkAccelerationStructureInstanceKHR {
    VkTransformMatrixKHR          transform;
    uint32_t                      instanceCustomIndex:24;
    uint32_t                      mask:8;
    uint32_t                      instanceShaderBindingTableRecordOffset:24;
    VkGeometryInstanceFlagsKHR    flags:8;
    uint64_t                      accelerationStructureReference;
} VkAccelerationStructureInstanceKHR;

The 24-bit value written to instanceCustomIndex on the host side directly populates gl_InstanceCustomIndexEXT in the shader. Because it is a 24-bit field, the maximum value it can represent is 16,777,215 (0xFFFFFF).

Supported Shader Stages

The variable is accessible in the ray tracing pipeline stages that process intersections:

  • Intersection Shaders (.rint): Useful if custom primitive intersection logic depends on instance-specific properties.
  • Any-Hit Shaders (.rahit): Commonly used to look up material data to evaluate alpha-testing or opacity cutouts.
  • Closest-Hit Shaders (.rchit): Frequently used to read material properties, transform data, and lighting parameters to shade the final hit point.

It is not available in ray generation, miss, or callable shaders, as these stages do not execute in the direct context of an intersected geometry instance.

gl_InstanceCustomIndexEXT vs. gl_InstanceID

GLSL ray tracing provides two distinct instance identifiers:

  • gl_InstanceID: An automatically generated, zero-based index representing the position of the instance within the TLAS array. This value changes if instances are reordered or compacted.
  • gl_InstanceCustomIndexEXT: A persistent, user-defined value that remains constant regardless of how instances are arranged or ordered inside the acceleration structure.

Practical Use Cases

Because ray tracing pipelines handle arbitrary scenes with diverse materials, shaders require a way to access per-instance resources without relying on individual draw calls. Common applications of gl_InstanceCustomIndexEXT include:

  1. Material Indexing: Mapping the hit instance directly to an array of material definitions stored in a Shader Storage Buffer Object (SSBO).
  2. Bindless Descriptor Indexing: Indexing into descriptor arrays containing textures, vertex buffers, or index buffers associated with that specific model instance.
  3. Object Identification: Passing an internal entity ID or mesh identifier back through the ray payload for physics queries, picking, or visibility checks.