Meshlet Normal Cone Culling in GLSL Task Shaders

Normal cone culling in GLSL task shaders provides a high-throughput method for discarding back-facing geometry at the cluster level before vertex processing and rasterization occur. By calculating a representative bounding cone for all triangle surface normals within a meshlet, a task shader can test the entire cluster against the camera's view position using a single geometric comparison. If the cone indicates that every triangle within the meshlet faces away from the camera, the task shader simply drops the meshlet, preventing subsequent mesh shader invocations and eliminating wasted vertex shading and rasterization work.

Understanding the Meshlet Normal Cone

A meshlet is a small, localized cluster of geometry typically containing up to 64 vertices and 128 primitives. Instead of testing the orientation of each individual triangle on the rasterizer, precomputed spatial and normal bounds represent the entire cluster:

  1. Cone Apex / Position: A spatial anchor point representing the cluster's origin or the apex from which normals diverge.
  2. Cone Axis: A normalized vector (\(\vec{A}\)) representing the average orientation of all triangle normals inside the meshlet.
  3. Cone Cutoff (Half-Angle): The angle \(\alpha\) representing the maximum divergence of any triangle normal from the cone axis \(\vec{A}\). The cosine or sine of this half-angle is usually precalculated and packed as a scalar value.
       \       /
        \  A  /
         \ | /  <- Cone Half-Angle (α)
          \|/
         Apex

The Mathematical Culling Test

For an individual triangle with normal \(\vec{N}\) and position \(\vec{P}\), the triangle is back-facing relative to a camera at position \(\vec{C}\) when:

\[(\vec{P} - \vec{C}) \cdot \vec{N} \ge 0\]

Assuming \(\vec{V} = \text{normalize}(\vec{P} - \vec{C})\) is the normalized view vector pointing from the camera toward the surface, back-facing occurs when \(\vec{V} \cdot \vec{N} \ge 0\).

When extending this logic to a normal cone with axis \(\vec{A}\) and half-angle \(\alpha\), the worst-case normal points away from the view vector by at most \(\alpha\). The entire meshlet is guaranteed to be back-facing if the minimum dot product between any normal inside the cone and the view vector remains greater than or equal to zero.

Using trigonometric identity, the back-face condition for the entire cone is:

\[\vec{V} \cdot \vec{A} \ge \sin(\alpha)\]

If this inequality holds true, every possible normal vector within the cone angle forms an angle greater than or equal to 90 degrees relative to the incoming view direction, meaning no triangle in the meshlet can face the camera.

Implementation in GLSL Task Shaders

Task shaders (GL_EXT_mesh_shader) execute in workgroups where each thread can evaluate one or more meshlets. The shader loads the precomputed bounding data, computes the view vector dynamically, and conditionally emits mesh shader workloads.

#version 460
#extension GL_EXT_mesh_shader : require

layout(local_size_x = 32) in;

struct MeshletBounds {
    vec3 coneApex;
    vec3 coneAxis;
    float coneCutoff; // Precalculated as sin(alpha)
};

layout(std430, binding = 0) readonly buffer MeshletBoundsBuffer {
    MeshletBounds bounds[];
};

uniform vec3 u_CameraPosition;

shared uint s_VisibleMeshletCount;

void main() {
    uint meshletIndex = gl_GlobalInvocationID.x;
    
    // Read precomputed bounding cone
    MeshletBounds b = bounds[meshletIndex];
    
    // Vector from the camera to the meshlet cone apex
    vec3 viewDir = normalize(b.coneApex - u_CameraPosition);
    
    // Evaluate normal cone culling condition
    bool isBackFacing = dot(viewDir, b.coneAxis) >= b.coneCutoff;
    
    // Compact visible meshlets across the workgroup
    uvec4 ballot = subgroupBallot(!isBackFacing);
    uint localIndex = subgroupBallotExclusiveBitCount(ballot);
    
    if (gl_SubgroupInvocationID == 0) {
        s_VisibleMeshletCount = subgroupBallotBitCount(ballot);
    }
    barrier();
    
    // Allocate mesh shader invocations only for surviving meshlets
    EmitMeshTasksEXT(s_VisibleMeshletCount, 1, 1);
}

Performance Advantages Over Fixed-Function Culling

Traditional back-face culling occurs in hardware after vertices are transformed, assembled into primitives, and projected into screen space. In contrast, normal cone culling inside a task shader provides several performance benefits:

  • Eliminates Shading Overhead: Primitives are culled before the mesh shader executes, meaning vertex attribute transformations, skinning, and dynamic tessellation calculations are never run for occluded clusters.
  • Reduces Index Buffer Processing: Meshlets that fail the cone test do not submit primitive connectivity data down the pipeline.
  • Amortizes Costs: A single dot product and scalar comparison in the task shader evaluates dozens of triangles simultaneously, drastically reducing the overall workload on hardware culling units.