Mapping Geometry to GLSL Hit Shaders with Hit Groups

In hardware-accelerated ray tracing pipelines such as Vulkan using GLSL, hit groups act as the organizational bridge linking scene geometry to the corresponding shader execution logic. This article explains how hit group configurations connect different geometry types—such as built-in triangle meshes and custom procedural primitives—to specific GLSL hit shaders via the Shader Binding Table (SBT), enabling flexible and performant material and intersection processing.

What Is a Hit Group?

A hit group is a pipeline construct that bundles up to three specific ray tracing shaders into a single dispatch unit:

  • Intersection Shader: Calculates ray intersections for non-standard or procedural shapes.
  • Any-Hit Shader: Executes upon any valid intersection along a ray's path, commonly used for alpha testing and transparency.
  • Closest-Hit Shader: Executes once for the nearest confirmed intersection, typically evaluating surface materials, textures, and secondary lighting.

Hit groups are defined during ray tracing pipeline creation and referenced as discrete records within the Shader Binding Table (SBT). When a ray hits an object, the GPU resolves the hit to one specific hit group record.

Mapping Distinct Geometry Types

Modern ray tracing APIs separate geometry into two main categories, each requiring a specific hit group configuration to execute properly:

1. Triangle Geometry

Triangle meshes utilize dedicated fixed-function ray-triangle intersection hardware. Because intersection calculations are handled automatically by the GPU:

  • The Intersection Shader slot remains unused (or null).
  • The hit group relies solely on an optional Any-Hit Shader (for discarded fragments/transparency) and a Closest-Hit Shader (for surface evaluation).

2. Procedural Geometry (AABBs)

Procedural primitives—such as spheres, hair strands, or signed distance fields—are represented in the Bottom-Level Acceleration Structure (BLAS) as Axis-Aligned Bounding Boxes (AABBs). Because hardware cannot natively evaluate these shapes:

  • An Intersection Shader is mandatory. It contains user-defined GLSL code that tests the ray against the mathematical shape and calls reportIntersectionEXT().
  • The hit group combines this custom intersection logic with an Any-Hit and a Closest-Hit Shader to process successful intersections reported by the custom routine.

The Shader Binding Table (SBT) Resolution

The mapping between a geometric instance and its hit group is resolved dynamically via indexing equations during traversal. In GLSL, when calling traceRayEXT(), indexing into the SBT hit group records is determined by combining several variables:

\[\text{Hit Group Index} = \text{Ray Offset} + (\text{Geometry Index} \times \text{Ray Stride}) + \text{Instance SBT Offset}\]

  • Instance SBT Offset: Defined on the acceleration structure instance, pointing to a base block of hit groups assigned to that object.
  • Geometry Index: Assigned to individual geometries within a single BLAS, allowing a multi-part mesh containing both triangles and AABBs to point to distinct hit groups.
  • Ray Offset and Stride: Passed directly in the GLSL traceRayEXT() call, allowing different ray types (such as primary visibility rays versus shadow rays) to invoke completely different hit groups for the exact same geometry.

Significance in Pipeline Architecture

Hit group configurations completely decouple geometric representations from rendering techniques. By allowing combinations of intersection, any-hit, and closest-hit shaders to be addressed per geometry and per ray type, developers can mix hardware-accelerated triangles and custom analytical shapes within the same acceleration structure, while ensuring each geometry receives the exact mathematical testing and shading routine it requires.