Any-Hit Shaders for Alpha Testing in GLSL Ray Tracing

In hardware-accelerated ray tracing with GLSL, handling transparent cutouts such as foliage, chain-link fences, and decals requires dynamic evaluation of surface opacity during ray traversal. The any-hit shader stage fulfills this role by intercepting potential ray-primitive intersections before they are committed as the closest hit. By sampling the surface texture's alpha channel inside the any-hit shader, the pipeline can conditionally discard non-opaque fragments, allowing the ray to continue along its path seamlessly.

The Role of the Any-Hit Shader in Traversal

During acceleration structure traversal, the ray tracing hardware identifies candidate intersections with geometric primitives. For fully opaque geometry, the traversal hardware automatically tracks the closest intersection without executing intermediate user shading logic. However, when geometry contains transparent cutouts, geometric triangles do not perfectly match the visible shape.

The any-hit shader sits between primitive intersection testing and the recording of a valid hit. When the ray intersects a primitive belonging to non-opaque geometry, the traversal pauses, and the any-hit shader is invoked. This stage determines whether the intersection is physically valid or if it represents an invisible portion of the texture.

Implementing Alpha Cutouts in GLSL

To execute alpha testing, the any-hit shader reconstructs the surface attributes at the hit point and evaluates the texture alpha against a cutoff threshold.

  1. Attribute Interpolation: The shader receives barycentric coordinates through hitAttributeEXT or built-in variables. Using these coordinates, it fetches and interpolates vertex data (such as texture coordinates) from storage buffers.
  2. Texture Sampling: The shader samples the diffuse or opacity texture using the interpolated UV coordinates.
  3. Alpha Evaluation: If the sampled alpha value falls below a predetermined cutoff threshold (e.g., alpha < 0.5), the intersection must be discarded.

The primary control function in GLSL for discarding hits is ignoreIntersectionEXT().

#version 460
#extension GL_EXT_ray_tracing : require

hitAttributeEXT vec2 hitAttribs;

layout(binding = 0, set = 0) buffer Vertices { vec4 v[]; } vertices;
layout(binding = 1, set = 0) buffer Indices { uint i[]; } indices;
layout(binding = 2, set = 0) uniform sampler2D alphaTexture;

void main()
{
    // Retrieve barycentric coordinates
    vec3 barycentrics = vec3(1.0 - hitAttribs.x - hitAttribs.y, hitAttribs.x, hitAttribs.y);

    // Interpolate UV coordinates from buffer data
    vec2 uv = interpolateUV(barycentrics);

    // Sample alpha
    float alpha = texture(alphaTexture, uv).a;

    // Discard intersection if below threshold
    if (alpha < 0.5)
    {
        ignoreIntersectionEXT();
    }
}

When ignoreIntersectionEXT() is invoked, the ray tracing pipeline rejects the candidate hit. The ray continues traversing along its trajectory with its maximum distance (gl_RayTmaxEXT) unchanged, ignoring the current primitive. If the alpha condition passes, the shader completes without calling the ignore function, and the pipeline accepts the intersection, updating gl_RayTmaxEXT to this hit point.

Acceleration Structure and Pipeline Setup

For the any-hit shader to execute, the corresponding geometry instances in the Bottom-Level Acceleration Structure (BLAS) must not be flagged with gl_GeometryOpaqueEXT (or VK_GEOMETRY_OPAQUE_BIT_KHR). Marking geometry as opaque instructs the hardware to bypass the any-hit stage completely for performance optimization.

Meshes requiring alpha testing must be marked with gl_GeometryNoDuplicateAnyHitInvocationEXT to prevent the any-hit shader from running multiple times for the same primitive, while purely solid meshes should retain the opaque flag.

Performance Considerations

Invoking any-hit shaders introduces significant performance overhead compared to fixed-function ray traversal, as it forces the ray tracing cores to switch context to the general execution units. To maximize throughput:

  • Isolate Alpha Geometry: Keep transparent cutouts in separate BLAS geometries from solid meshes to minimize the number of rays invoking any-hit shaders.
  • Shadow Rays: When tracing shadow rays against cutout geometry, call terminateRayEXT() immediately after confirming an opaque hit (where alpha >= threshold) to stop further traversal early.
  • Minimize Operations: Avoid complex lighting calculations, dynamic branching, or unnecessary memory loads inside the any-hit shader; restrict the execution strictly to texture sampling and the alpha comparison.