Meshlet Frustum Culling with GLSL Task Shaders

Frustum culling on meshlets using task shaders allows the GPU to discard non-visible geometry clusters before reaching the rasterizer, substantially reducing primitive processing overhead. By testing the bounding volume of each meshlet against the camera's view frustum directly within the task shader, the graphics pipeline can dynamically determine which mesh shaders to spawn. This article details the mechanics of meshlet-level culling, how task shaders process geometry batches, and the GLSL implementation required to discard culled meshlets efficiently using subgroup operations.

The Role of Task Shaders in Meshlet Culling

In the mesh shading pipeline, the task shader acts as an amplification and culling stage that runs prior to the mesh shader. Geometry is pre-processed into meshlets—small clusters of vertices and primitives (typically up to 64 vertices and 128 triangles).

Rather than sending an entire draw call to the mesh shader, the task shader executes over a collection of meshlets. Each invocation or workgroup reads the spatial bounds of a meshlet, tests those bounds against the camera's view frustum, and determines visibility. If a meshlet falls outside the frustum, it is discarded. The task shader then allocates mesh shader workgroups only for the surviving meshlets using the built-in emission function.

Implementing the Frustum Test

Each meshlet contains precomputed bounding geometry, such as an Axis-Aligned Bounding Box (AABB) or an oriented bounding sphere. Spheres are often preferred for their low computational overhead:

  1. Frustum Representation: The view frustum is represented as six planes, each defined by an outward- or inward-pointing normal and a distance scalar \((Ax + By + Cz + D = 0)\).
  2. Sphere-Plane Distance: The signed distance from the sphere center to each of the six planes is calculated: \[\text{distance} = \mathbf{plane.xyz} \cdot \mathbf{sphere.center} + \text{plane.w}\]
  3. Visibility Evaluation: If the signed distance is less than \(-\text{radius}\) for any of the six planes, the bounding sphere is completely outside the frustum, and the meshlet is culled.

Compaction with Subgroup Operations

Because task shaders can cull a variable number of meshlets across a workgroup, sparse outputs must be compacted so that the spawned mesh shaders receive contiguous payloads without idle threads. This is done using GLSL subgroup operations (warp/wavefront intrinsics):

  1. Subgroup Ballot: Threads within a workgroup evaluate their assigned meshlet's visibility. The results are gathered into a bitmask using subgroupBallot().
  2. Prefix Sum (Compaction): Each visible meshlet computes its compact index within the surviving group using subgroupBallotExclusiveBitCount().
  3. Payload Population: The visible meshlet indices are written into the shared taskPayloadSharedEXT variable at their compacted positions.
  4. Task Emission: The first thread in the subgroup calls EmitMeshTasksEXT() with the count of surviving meshlets obtained from subgroupBallotBitCount().

GLSL Implementation Example

The following GLSL code demonstrates frustum culling on a 32-meshlet workgroup using the GL_EXT_mesh_shader and GL_KHR_shader_subgroup_ballot extensions.

#version 460
#extension GL_EXT_mesh_shader : require
#extension GL_KHR_shader_subgroup_ballot : require

layout(local_size_x = 32, local_size_y = 1, local_size_z = 1) in;

struct MeshletBounds {
    vec3 center;
    float radius;
};

layout(std430, binding = 0) readonly buffer MeshletBoundsBuffer {
    MeshletBounds bounds[];
};

layout(binding = 1) uniform CameraData {
    mat4 viewProj;
    vec4 frustumPlanes[6];
};

struct TaskPayload {
    uint meshletIndices[32];
};

taskPayloadSharedEXT TaskPayload s_payload;

bool isSphereInFrustum(vec3 center, float radius) {
    for (int i = 0; i < 6; ++i) {
        if (dot(frustumPlanes[i].xyz, center) + frustumPlanes[i].w < -radius) {
            return false;
        }
    }
    return true;
}

void main() {
    uint globalIndex = gl_GlobalInvocationID.x;
    uint localIndex  = gl_LocalInvocationID.x;

    // Evaluate visibility
    MeshletBounds b = bounds[globalIndex];
    bool visible = isSphereInFrustum(b.center, b.radius);

    // Vote across the subgroup to find surviving meshlets
    uvec4 ballot = subgroupBallot(visible);
    uint totalVisible = subgroupBallotBitCount(ballot);

    // If visible, write into compacted payload array
    if (visible) {
        uint compactIndex = subgroupBallotExclusiveBitCount(ballot);
        s_payload.meshletIndices[compactIndex] = globalIndex;
    }

    // Emit mesh shader workgroups only for visible meshlets
    if (localIndex == 0) {
        EmitMeshTasksEXT(totalVisible, 1, 1);
    }
}

Execution Behavior

When totalVisible evaluates to zero, EmitMeshTasksEXT(0, 1, 1) halts further processing for that workgroup, bypassing the mesh shader entirely. Surviving workgroups launch the exact number of mesh shaders required, where gl_WorkGroupID.x in the mesh shader indexes directly into s_payload.meshletIndices to fetch and render the visible meshlet data.