Understanding layout(max_vertices = N) in GLSL

In GLSL, the layout(max_vertices = N) declaration specifies the maximum number of vertices that a single invocation of a geometry shader can emit. This article explains how graphics hardware uses this upper bound to allocate on-chip memory, how the declaration enforces execution boundaries, and why setting an accurate value is critical for GPU pipeline performance.

The Role of max_vertices = N

A geometry shader processes an entire input primitive (such as a point, line, or triangle) and can emit zero or more primitives as output. Because the number of output vertices can vary dynamically per invocation, the GPU driver needs an explicit maximum limit to manage memory allocation.

The declaration is defined on the output layout in the shader source:

layout(triangle_strip, max_vertices = N) out;

Here, N represents a positive integer constant representing the hard upper limit of vertices the shader is allowed to write using EmitVertex().

Memory Allocation Limits on the GPU

Graphics hardware processes geometry shader invocations concurrently across multiple streaming multiprocessors or compute units. To enable parallel execution, the hardware must reserve storage space for the output of each thread before the shader begins running.

When you specify max_vertices = N, the GPU calculates the required memory per invocation:

\[\text{Reserved Memory} = N \times \text{Size of All Output Attributes}\]

This storage must accommodate all built-in outputs (such as gl_Position) and user-defined out variables for \(N\) vertices. This memory is typically allocated in fast, on-chip scratchpad memory or registers.

Performance and Occupancy Implications

The value of \(N\) directly affects shader occupancy—the number of active, concurrent threads a GPU core can execute at a given time:

  • Overestimating \(N\): If you define \(N = 128\) but your shader only outputs 3 vertices, the driver must still allocate sufficient memory for 128 full vertices per invocation. This wasted capacity severely reduces the number of active threads that can fit into on-chip storage, stalling GPU execution units and reducing overall frame rates.
  • Underestimating \(N\): If the shader calls EmitVertex() more than \(N\) times during execution, any vertex emitted beyond the limit \(N\) is silently discarded by the implementation. This leads to missing geometry and visual artifacts.

Best Practices

To achieve optimal performance when working with geometry shaders in GLSL:

  1. Keep \(N\) as small as possible: Set \(N\) to the exact mathematical maximum number of vertices the logic can emit.
  2. Minimize per-vertex output attributes: Because total buffer size is \(N \times \text{vertex size}\), reducing output attributes proportionally lowers on-chip memory usage.
  3. Avoid high amplification factors: Geometry shaders are optimized for low-volume amplification (e.g., turning points into quads or calculating face normals). If \(N\) needs to be large (tens or hundreds of vertices), use tessellation evaluation shaders, compute shaders, or modern mesh shaders instead.