Bounding Sphere Frustum Culling in GLSL Task Shaders

This article details how to implement bounding sphere frustum culling inside a GLSL task shader to eliminate non-visible geometry clusters prior to mesh shader dispatch. By running a signed distance plane-sphere intersection test on the GPU workgroup level, developers can dynamically reject entire meshlets before launching expensive mesh generation workloads.

The Core Algorithm: Signed Distance Plane-Sphere Test

The primary algorithm used to evaluate bounding sphere frustum culling inside a GLSL task shader is the Signed Distance Plane-Sphere Intersection Test using planes defined in Hesse normal form.

A view frustum is defined by six bounding planes: left, right, bottom, top, near, and far. Each plane is represented as a normalized 4D vector:

\[\mathbf{P} = (A, B, C, D)\]

where \((A, B, C)\) represents the unit normal vector pointing toward the interior of the frustum, and \(D\) is the distance from the origin.

Given a bounding sphere with center point \(\mathbf{C} = (x, y, z)\) and radius \(r\), the signed distance \(d\) from the sphere center to a plane is calculated using a dot product:

\[d = (\mathbf{P}_{xyz} \cdot \mathbf{C}) + \mathbf{P}_w\]

The culling condition is evaluated as follows:

If \(d < -r\), the sphere lies entirely outside the plane and is completely outside the frustum (culled).
If \(d > r\), the sphere lies entirely inside the half-space of the plane.
If \(|d| \le r\), the sphere intersects the plane.

If the sphere falls outside (\(d < -r\)) for any of the six frustum planes, the corresponding geometry is invisible and discarded.

Implementation in GLSL Task Shaders

Task shaders (GL_EXT_mesh_shader or GL_NV_mesh_shader) execute in workgroups where each invocation typically processes an individual meshlet or instance.

1. Frustum Plane Transformation

To minimize computational overhead, the six frustum planes can be transformed into the local object space of the mesh during uniform setup. This eliminates the need to transform every bounding sphere center by the model-view matrix inside the shader invocation.

2. GLSL Culling Function

Inside the task shader, the culling test can be implemented as a loop over the six planes:

bool isSphereInFrustum(vec3 center, float radius, vec4 planes[6]) {
    for (int i = 0; i < 6; ++i) {
        if (dot(planes[i].xyz, center) + planes[i].w < -radius) {
            return false; // Culled
        }
    }
    return true; // Visible
}

3. Workgroup Compaction and Task Dispatch

After evaluating visibility, task shader threads must compact the visible indices before issuing the dispatch:

Each thread evaluates isSphereInFrustum for its assigned meshlet.
The workgroup uses GLSL subgroup operations (such as subgroupBallot and subgroupBallotBitCount) to determine the total number of surviving meshlets and compute local compaction offsets.
Visible meshlet IDs are written into a shared payload array.
Thread 0 calls EmitMeshTasksEXT(survivingCount, 1, 1) to spawn only the exact number of mesh shader invocations required for visible geometry.