Instance Masks in DirectX Raytracing

This article provides an overview of instance masks in DirectX Raytracing (DXR), explaining their architecture and role in ray traversal. It covers how instance masks are defined within acceleration structures, the mathematical bitwise logic used to evaluate them during ray dispatch, and how developers leverage them to cull unnecessary ray-primitive intersection tests directly in hardware to maximize rendering performance.

What Are Instance Masks?

In DirectX Raytracing, scene geometry is organized into a two-level hierarchy of acceleration structures: Bottom-Level Acceleration Structures (BLAS), which hold actual mesh geometry (triangles or procedural primitives), and Top-Level Acceleration Structures (TLAS), which contain instances of these BLAS along with transform matrices and rendering metadata.

An instance mask is an 8-bit integer (ranging from 0x00 to 0xFF) assigned to each instance inside the TLAS via the D3D12_RAYTRACING_INSTANCE_DESC structure:

typedef struct D3D12_RAYTRACING_INSTANCE_DESC {
    ...
    UINT InstanceMask : 8;
    ...
} D3D12_RAYTRACING_INSTANCE_DESC;

This 8-bit value acts as an identifier or category tag for each instance in the scene, categorizing objects into user-defined groups such as opaque geometry, transparent surfaces, shadow casters, or view-dependent assets.

Filtering Ray-Primitive Intersections

Instance masks filter ray-primitive intersections by determining whether a ray can enter a particular BLAS during the TLAS traversal phase. This filtering is resolved early in hardware, well before any triangle intersection tests or Any-Hit shader evaluations occur.

The Bitwise Mask Test

When launching a ray in HLSL via the TraceRay() intrinsic function, the developer specifies an 8-bit inclusion mask (InstanceInclusionMask) alongside ray flags, origins, and directions:

TraceRay(
    AccelerationStructure,
    RayFlags,
    InstanceInclusionMask, // 8-bit mask passed to the ray
    RayContributionToHitGroupIndex,
    MultiplierForGeometryContributionToHitGroupIndex,
    MissShaderIndex,
    Ray,
    Payload
);

During TLAS traversal, the GPU performs a bitwise AND operation between the ray's InstanceInclusionMask and the target instance’s InstanceMask:

\[\text{Test} = (\text{RayInclusionMask} \ \& \ \text{InstanceMask}) \neq 0\]

• Non-Zero Result: If the result of the bitwise AND operation is greater than zero, the test passes. The ray continues its traversal into the child BLAS associated with that instance to test for bounding volume and primitive intersections.
• Zero Result: If the result is zero, the test fails. The GPU completely ignores the instance. The entire BLAS attached to that instance is culled from the ray's traversal path.

Why Instance Masks Are Essential

Hardware-Level Early Out

Because instance mask evaluation takes place at the TLAS level, instances that fail the mask test incur zero primitive intersection costs. The hardware bypasses all BVH node traversals, triangle tests, Any-Hit shaders, and Closest-Hit shaders that would have otherwise executed for that instance.

Specialized Ray Pipelines

By allocating different bits of the 8-bit mask to different categories, developers can finely tune ray behavior for specific rendering passes:

• Shadow Rays: A shadow ray can use a mask that only tests against shadow-casting geometry, skipping non-shadow-casting particles or decals entirely.
• Reflection Rays: Rough reflections can omit fine-detail geometry like foliage or wireframes to save traversal time.
• First-Person Assets: Player character models can be masked out from camera rays to avoid self-clipping while remaining visible to shadow rays cast by external lights.
• Layered Visibility: Transparent geometry can be isolated from opaque visibility sweeps, preventing costly Any-Hit shader invocations during primary visibility checks.