DirectX Meshlets: Frustum and Backface Culling

This article explores the concept of meshlets within modern graphics pipelines, specifically focusing on DirectX 12. It covers the structural definition of meshlets, how spatial and orientation bounding volumes are generated for them, and the algorithmic process used by Amplification and Mesh Shaders to perform rapid cluster-level frustum and backface culling before triangles ever reach the rasterizer.

What Are Meshlets?

A meshlet is a localized sub-mesh representing a partition of a larger 3D geometric model. In traditional rendering, an entire index buffer is dispatched to the Input Assembler. With the DirectX 12 Mesh Shader pipeline, models are pre-processed offline or during asset cooking into small, discrete clusters of vertices and unique primitives.

Typically, a meshlet is constrained to a fixed maximum size—often up to 64 vertices and 126 triangles, designed to fit cleanly into graphics hardware thread groups (such as 32-thread warps or 64-thread wavefronts). Because each meshlet acts as an independent, self-contained mini-mesh with its own local index list, the GPU can process them with high parallelism, avoiding the cache contention and vertex-reuse limits inherent in classic index buffers.

Bounding Meshlets for Culling

To cull non-visible geometry as early as possible, each meshlet is tagged with precomputed metadata detailing its physical and angular boundaries. There are two primary bounding mechanisms assigned to a meshlet:

  1. Spatial Bounding Volumes: A bounding sphere or an Axis-Aligned Bounding Box (AABB) that encapsulates all vertex positions belonging to the meshlet. Bounding spheres (defined by a 3D center position and a radius) are commonly preferred due to their rotation invariance and the minimal mathematical operations required for collision testing.
  2. Normal Cones (Orientation Bounds): A bounding cone that encapsulates all the face normals of the triangles within the meshlet. The cone is defined by:
    • An apex (origin point, often coinciding with the bounding sphere center or an optimized cluster center).
    • An axis unit vector pointing in the average direction of the cluster's normals.
    • A cutoff angle or cosine threshold representing the maximum angular spread of any surface normal relative to the cone axis.

Frustum Culling

Frustum culling discards geometry located outside the camera's view volume. In DirectX 12, this is typically handled in the Amplification Shader (formerly called the Task Shader) before launching Mesh Shaders:

  • The Test: The bounding sphere of the meshlet is transformed into view space or world space and tested against the six planar equations of the camera's view frustum.
  • The Decision: If the signed distance from the sphere center to any of the six planes is less than the negative radius of the sphere, the sphere sits entirely outside the frustum.
  • The Execution: If culled, the Amplification Shader suppresses the dispatch of the corresponding Mesh Shader threadgroup entirely. Vertices are never unpacked, indices are never read, and primitive shading is skipped completely.

Cluster-Level Backface Culling

Traditional pipelines rely on the hardware rasterizer to cull individual back-facing triangles after vertex transformation. Meshlets allow cluster-level backface culling, eliminating dozens of triangles at once using the normal cone:

  • The Test: A vector is calculated from the camera position to the meshlet's bounding center. The dot product between this view vector and the cone axis is evaluated.
  • The Decision: If the angle between the view vector and the cone axis indicates that even the most extreme normal in the cone still faces away from the camera, every triangle in that meshlet is guaranteed to be back-facing.
  • The Execution: If the dot product is greater than the cone’s spread threshold, the meshlet is entirely facing away from the viewer. The shader discards the entire meshlet immediately.

Performance Benefits in DirectX

Integrating frustum and backface culling into the DirectX Mesh Shader pipeline shifts geometry evaluation from fixed-function post-transform stages to early compute-like stages. By evaluating bounding spheres and normal cones hierarchically within Amplification Shaders, systems eliminate unneeded geometry dispatches, drastically reduce memory bandwidth, and preserve GPU rasterization capacity for visible surfaces.