Custom Volumes with GLSL Intersection Shaders

In hardware-accelerated ray tracing using GLSL, rendering non-triangular shapes requires procedural intersection handling. Instead of relying on built-in hardware ray-triangle testers, developers construct custom geometric bounding volumes within acceleration structures and resolve the exact hit points using intersection shaders. This article explains how Axis-Aligned Bounding Boxes (AABBs) act as the spatial bounds for custom geometry, how rays invoke GLSL intersection shaders, and how mathematical volume boundaries are evaluated and reported to the ray tracing pipeline.

Defining Procedural Geometry with AABBs

The ray tracing acceleration structure cannot natively interpret mathematical formulas such as implicit surfaces, signed distance fields (SDFs), or higher-order splines. To integrate custom shapes, the application defines Axis-Aligned Bounding Boxes (AABBs) at the API level (such as Vulkan via VkAccelerationStructureGeometryAabbsDataKHR).

Each procedural primitive is represented by minimum and maximum spatial coordinates:

  • minX, minY, minZ
  • maxX, maxY, maxZ

These coordinates define a conservative bounding volume that fully encloses the procedural object. During bottom-level acceleration structure (BLAS) generation, the hardware builds a bounding volume hierarchy (BVH) around these user-defined boxes.

Invoking the GLSL Intersection Shader

During ray traversal, the traversal hardware tests rays against the BVH nodes. When a ray intersects an AABB containing custom geometry, the pipeline pauses hardware traversal and invokes the corresponding GLSL intersection shader (typically compiled with the .rint extension under GL_EXT_ray_tracing).

Inside the intersection shader, the developer defines the exact analytical or numerical boundary of the geometry. The shader retrieves the ray parameters in object space using built-in variables:

  • gl_ObjectRayOriginEXT: The ray origin transformed into the primitive's local coordinate space.
  • gl_ObjectRayDirectionEXT: The ray direction in local space.
  • gl_RayTminEXT and gl_RayTmaxEXT: The valid distance interval along the ray.

Computing Ray-Volume Intersections

The intersection shader calculates the parameter \(t\) along the ray equation \(P(t) = \text{Origin} + t \cdot \text{Direction}\). For an analytical sphere centered at the origin with radius \(r\), the shader solves the quadratic equation:

\[\| \text{Origin} + t \cdot \text{Direction} \|^2 = r^2\]

For more complex surfaces, such as volumetric shapes or fractals, the shader executes ray marching algorithms to approximate the surface boundary within the bounding volume.

Reporting Hits to the Pipeline

If the ray intersects the custom surface within the bounding box and between gl_RayTminEXT and gl_RayTmaxEXT, the shader reports the hit back to the traversal engine using the built-in function:

bool reportIntersectionEXT(float hitT, uint hitKind);
  • hitT: The parametric distance along the ray where the intersection occurred.
  • hitKind: An unsigned integer identifying the type of hit (e.g., front-face, back-face, or a material ID) passed to Any-Hit and Closest-Hit shaders.

If the hit falls outside the valid parametric range or the ray misses the geometric shape inside the AABB, the shader completes without calling reportIntersectionEXT, and ray traversal resumes automatically.

Passing Custom Surface Attributes

Because custom geometries lack default barycentric coordinates, attributes such as surface normals or UV coordinates must be calculated within the intersection shader. Developers declare hit attribute variables using the hitAttributeEXT storage qualifier:

hitAttributeEXT vec3 hitNormal;

When reportIntersectionEXT returns true, the pipeline stores the current state of variables declared with hitAttributeEXT. These values become accessible to subsequent Closest-Hit and Any-Hit stages for lighting and shading calculations.

Ensuring Conservative Bounding Constraints

For intersection shaders to function accurately and performantly, the defining bounding volume must be strictly conservative:

  1. Under-bounding: If an AABB is smaller than the procedural surface, parts of the geometry extending outside the box will not trigger the intersection shader, causing clipping artifacts.
  2. Over-bounding: Excessively large AABBs force the pipeline to execute intersection shaders unnecessarily for rays that pass near, but not through, the intended geometry, degrading performance.