Primitive Restart in GLSL Mesh Shader Index Buffers

GLSL mesh shaders do not evaluate or support primitive restart indices within their output index buffers. Because the mesh shading pipeline generates discrete, unindexed primitive topologies directly within the workgroup rather than consuming continuous indexed strips, special restart tokens are treated as standard vertex indices rather than stream delimiters.

Topology and Discrete Primitive Output

In traditional fixed-function and vertex-driven pipelines, primitive restart allows indexed drawing commands (such as glDrawElements) to break triangle strips, line strips, or triangle fans into disconnected components using a designated sentinel index (typically 0xFFFFFFFF or 0xFFFF).

Mesh shaders, standardized through extensions such as GL_EXT_mesh_shader (and its Vulkan equivalent VK_EXT_mesh_shader), fundamentally alter this model. Mesh shaders declare their output topology statically in the shader layout qualifiers:

layout(triangles, max_vertices = N, max_primitives = M) out;

Valid output topologies are limited strictly to discrete primitives:

  • points
  • lines
  • triangles

Strip and fan topologies are not valid output types for mesh shaders. Because every primitive is explicitly defined as an isolated tuple—such as uvec3 for triangles or uvec2 for lines—the concept of breaking an ongoing strip with a restart token does not apply.

Index Buffer Representation and Built-ins

Mesh shaders write primitive connectivity into built-in array outputs, such as:

out uvec3 gl_PrimitiveTriangleIndicesEXT[];
out uvec2 gl_PrimitiveLineIndicesEXT[];
out uint  gl_PrimitivePointIndicesEXT[];

Each element in these arrays directly references a local vertex index in the shader’s per-vertex output arrays (such as gl_MeshVerticesEXT[]).

The hardware rasterizer treats the values in gl_PrimitiveTriangleIndicesEXT as direct lookups. If a shader writes 0xFFFFFFFF into an index tuple:

  1. The rasterizer does not intercept this value as a primitive restart marker.
  2. The value is interpreted literally as an index pointing to vertex 4294967295.
  3. Because the vertex index exceeds max_vertices, it results in an out-of-bounds read from the mesh shader's local vertex storage, yielding undefined primitive geometry or triggering robust buffer access clamping.

Discarding and Culling Primitives

Instead of using primitive restart to omit geometry or terminate sequences, mesh shaders control primitive generation dynamically through the primitive count parameter and individual index assignment:

  • Dynamic Primitive Count: Shaders call SetMeshOutputsEXT(vertexCount, primitiveCount) to set the exact number of active primitives emitted by the workgroup. Primitives outside this range are never generated or rasterized.
  • Per-Primitive Culling: To discard an individual primitive without altering contiguous ordering, developers can either pack active primitive indices to fill only the valid range up to primitiveCount or mark primitives using the gl_CullPrimitiveEXT[] per-primitive output boolean array.