gl_RayFlagsCullBackFacingTrianglesEXT in GLSL Ray Tracing

This article explains the purpose and mechanics of gl_RayFlagsCullBackFacingTrianglesEXT in GLSL ray tracing pipelines and ray queries. You will learn how this ray flag interacts with hardware intersection testing to discard back-facing geometry, how face orientation is determined, and how its use improves rendering performance and prevents artifacts in closed meshes.

What is gl_RayFlagsCullBackFacingTrianglesEXT?

In GLSL (specifically through extensions like GL_EXT_ray_tracing and GL_EXT_ray_query), gl_RayFlagsCullBackFacingTrianglesEXT is an integer bitmask flag passed into ray traversal functions such as traceRayEXT() or rayQueryInitializeEXT(). Its primary role is to instruct the ray traversal engine to automatically discard intersection candidates with triangles classified as back-facing relative to the ray direction.

How Intersection Filtering Works

When a ray traverses a bottom-level acceleration structure (BLAS), the hardware performs ray-triangle intersection tests. When gl_RayFlagsCullBackFacingTrianglesEXT is active:

  1. Winding Order Determination: The triangle’s orientation is evaluated based on the vertex winding order defined when the acceleration structure was built (such as VK_FRONT_FACE_COUNTER_CLOCKWISE or VK_FRONT_FACE_CLOCKWISE in the Vulkan API).
  2. Facing Test: The dot product between the geometric normal (derived from vertex order) and the ray's incoming direction vector is evaluated. If the surface normal points away from the ray's origin—meaning the ray hits the back side of the polygon—the triangle is classified as a back face.
  3. Culling: The traversal unit immediately ignores the hit. Neither any-hit shaders nor closest-hit shaders are invoked for that intersection, and the traversal engine continues searching for other potential hits along the ray.

Key Use Cases

  • Performance Optimization: Testing and resolving hits incurs shader execution overhead, especially if any-hit shaders are bound. Culling back faces eliminates roughly half of the candidate intersections on closed, manifold geometry.
  • Shadow and Ambient Occlusion Rays: For opaque, closed objects, tracing shadow rays with back-face culling prevents self-shadowing artifacts on internal surfaces and speeds up early-exit intersection logic.
  • Preventing Unwanted Internal Reflections: When calculating primary or indirect reflection rays, culling back faces ensures rays originating from exterior surfaces do not register hits against interior-facing geometry.

Behavior and Limitations

  • Triangle Primitives Only: This flag only affects built-in triangle geometry. It has no effect on procedural primitives defined by axis-aligned bounding boxes (AABBs) using intersection shaders.
  • Single-Sided vs. Double-Sided Materials: If geometry represents thin, single-layer surfaces (such as foliage, paper, or cloth), enabling this flag will make those surfaces invisible when viewed from behind.
  • Interaction with Other Flags: It can be combined with other traversal flags (such as gl_RayFlagsOpaqueEXT), but it is mutually exclusive with gl_RayFlagsCullFrontFacingTrianglesEXT. Setting both will cause the ray to cull all triangles, hitting nothing.