How executeCallableEXT Works in GLSL Ray Tracing

This article explains how the executeCallableEXT function executes a callable shader routine in GLSL under the GL_EXT_ray_tracing extension. It outlines the mechanics of the dynamic dispatch process, the mapping of the call to the Shader Binding Table (SBT), and how parameters are communicated between the caller and the invoked shader.

In modern ray tracing pipelines, executeCallableEXT allows ray generation, closest hit, miss, and other callable shaders to invoke execution blocks dynamically. Unlike hardcoded function calls, callable shaders are dispatched dynamically at runtime, behaving similarly to a virtual function call or an indirect jump within the GPU pipeline.

The function takes the following signature:

void executeCallableEXT(uint sbtRecordIndex, int callableDataLocation);

1. Shader Binding Table (SBT) Resolution

When executeCallableEXT is executed, the GPU uses the first argument, sbtRecordIndex, to locate the appropriate shader routine in the Shader Binding Table.

The pipeline's host setup defines a callable shader binding region via VkStridedDeviceAddressRegionKHR. When the shader executes:

The GPU calculates the memory address of the target shader handle using the formula: targetAddress = callableBaseAddress + (sbtRecordIndex * callableStride).
The shader record stored at that address contains the shader group handle corresponding to the compiled callable shader, along with any embedded local shader parameters.
Control flow transfers to the callable shader specified by this record.

2. Passing Data with Storage Qualifiers

The second argument, callableDataLocation, links input and output parameters between the invoking shader and the callable shader without using an explicit parameter list.

Data communication relies on custom storage qualifiers matched by location indices:

In the calling shader, data is declared using the callableDataEXT qualifier:

layout(location = 0) callableDataEXT MyDataBlock {
    vec4 parameter1;
    vec4 result;
} myData;

In the callable shader, the corresponding data is received using the callableDataInEXT qualifier at the identical location:

layout(location = 0) callableDataInEXT MyDataBlock {
    vec4 parameter1;
    vec4 result;
} receivedData;

The GPU passes an effective pointer to this memory payload. Any modifications made by the callable shader to variables marked with callableDataInEXT directly alter the payload defined in the caller.

3. Execution and Return Flow

Calling executeCallableEXT suspends execution of the calling shader. The GPU switches execution contexts to the target callable shader entry point (main()).

Once the callable shader reaches the end of its routine:

Execution control returns to the caller immediately after the executeCallableEXT instruction.
The caller can then read any modified values stored in the callableDataEXT structure.
Callable shaders can also recursively invoke other callable shaders using executeCallableEXT, provided the implementation's maximum call depth limit (maxRayDispatchInvocationCount / call depth settings) is not exceeded.