GLSL Mesh Shader layout(triangles) out Explained

In modern graphics pipelines utilizing mesh shaders, declaring geometry output format is essential for the GPU's rasterization hardware to reconstruct surfaces properly. This article explains the role of the layout(triangles) out; declaration in a GLSL mesh shader, detailing how it configures output primitive assembly, dictates vertex index formatting, and establishes the primitive topology sent down the rendering pipeline.

Defining the Output Primitive Topology

The declaration layout(triangles) out; specifies that the mesh shader workgroup emits discrete triangles as its final geometric output. In the mesh shader pipeline, a single workgroup produces a micro-mesh containing a local vertex list and an index list. Declaring triangles establishes that the fixed-function primitive assembly hardware must interpret the emitted primitive indices in groups of three, rasterizing each valid triplet as an independent triangle.

Primitive Index Assembly

When layout(triangles) out; is active, the shader must write indices that correspond to three vertices for every emitted primitive. Depending on the GLSL extension used (such as GL_EXT_mesh_shader or GL_NV_mesh_shader), this configuration dictates the structure of the built-in output primitive index array:

  • Under GL_EXT_mesh_shader, primitive indices are written to gl_PrimitiveTriangleIndicesEXT[], an array of 3-component unsigned integer vectors (uvec3). Each element of this array defines the three local vertex indices required to assemble one triangle.
  • The indices point directly into the workgroup's local vertex array (such as gl_MeshVerticesEXT[]), with values ranging from 0 to max_vertices - 1.

Because the topology is explicitly set to independent triangles, the hardware does not assemble triangle strips or triangle fans; every individual primitive requires its own distinct set of three indices.

Interaction with Execution Limits and Primitive Counts

The triangles layout qualifier is combined with size declarations that define the maximum capacity of the mesh shader:

layout(triangles) out;
layout(max_vertices = N, max_primitives = M) out;

In this context:

  • max_primitives sets the upper bound for the number of triangles the workgroup can generate.
  • The actual number of assembled triangles is dynamically set at runtime using the built-in function SetMeshOutputsEXT(numVertices, numPrimitives).
  • The primitive assembler will only read up to numPrimitives elements from the index array, assembling precisely that many three-vertex triangles for rasterization.

Downstream Pipeline Effects

Once the mesh shader completes execution, the primitive assembly unit gathers the vertex attributes defined by the primitive index array. With triangles specified:

  1. Clipping and Culling: Fixed-function stages perform back-face culling, depth clipping, and frustum culling on a per-triangle basis using the three assembled vertices.
  2. Rasterization: The rasterizer interpolates per-vertex outputs across the barycentric coordinates of each triangle.
  3. Per-Primitive Attributes: Any attributes qualified with perprimitiveEXT (such as primitive IDs or layer selections) are mapped directly to each assembled triangle as a single entity rather than per vertex.