Mesh Shader Culling with gl_CullPrimitiveEXT

In modern GPU pipelines using the GL_EXT_mesh_shader extension, the gl_CullPrimitiveEXT built-in output array provides a direct, hardware-supported mechanism to discard individual primitives prior to rasterization. By setting a boolean flag per primitive directly inside the mesh shader, developers can dynamically cull back-facing, degenerate, or frustum-clipped geometry without needing to reorganize indices or compact output buffers, significantly improving rendering efficiency.

Declaration and Nature of the Built-In

When writing a GLSL mesh shader, enabling the GL_EXT_mesh_shader extension implicitly provides the per-primitive output array:

perprimitiveEXT out bool gl_CullPrimitiveEXT[];

The array size corresponds to the maximum number of output primitives declared in the shader's layout qualifier (e.g., layout(max_primitives = N) out;). Because it is declared with the perprimitiveEXT qualifier, each element in the array maps directly to a specific primitive generated by the mesh shader invocation.

How Primitive Discarding Operates

To discard a primitive, a thread in the mesh shader workgroup writes a boolean value to the gl_CullPrimitiveEXT array using that primitive's local index:

• gl_CullPrimitiveEXT[primIndex] = true;: Marks the primitive for discarding. The fixed-function hardware drops the primitive immediately after the mesh shader completes, preventing it from proceeding to clipping, setup, or the fragment shader.
• gl_CullPrimitiveEXT[primIndex] = false;: Keeps the primitive active for standard rasterization and fragment processing.

This evaluation happens on a per-primitive basis independently of vertex assembly. Even though the vertices belonging to a culled primitive may have been processed and output by the mesh shader, the primitive assembly stage bypasses any triangle or line marked with true.

Compaction Versus Individual Culling

Prior to primitive culling flags, eliminating invisible geometry within a mesh workgroup required manual thread compaction. Developers had to dynamically decrement the output count passed to SetMeshOutputsEXT() and re-index active primitives into contiguous memory slots to avoid rendering empty or degenerate triangles.

gl_CullPrimitiveEXT removes the need for workgroup-level stream compaction. The mesh shader can retain a static topology and assign fixed primitive index mappings per thread. When a thread computes that its assigned primitive is facing away from the camera, off-screen, or zero-area, it simply writes true to its index in gl_CullPrimitiveEXT. The underlying hardware pipeline handles the sparse primitive stream natively, reducing register pressure and avoiding expensive intra-workgroup barriers.