Understanding layout(vertices = N) out in GLSL

In OpenGL's programmable pipeline, the statement layout(vertices = N) out; in a Tessellation Control Shader (TCS) specifies the number of vertices present in the generated output patch. This article explains how this declaration dictates the shader's invocation count, manages per-patch output array sizing, and coordinates data flow to the subsequent Tessellation Evaluation Shader (TES).

The primary purpose of layout(vertices = N) out; is to define the size of the output patch produced by the Tessellation Control Shader. While the input patch size is determined on the CPU side using glPatchParameteri(GL_PATCH_VERTICES, ...), the TCS has the ability to change the topology by defining an entirely different number of output control points represented by N. For example, a TCS can receive a patch of 4 input vertices and output a patch of 16 control points for a bicubic Bézier surface by setting N to 16.

This declaration directly controls the execution model of the shader. The GPU executes the TCS once for every output vertex in the patch, meaning the shader is invoked N times per patch. Within the shader, the built-in variable gl_InvocationID specifies the index of the output vertex currently being processed, ranging from 0 to N - 1.

Additionally, setting the output vertex count sizes all per-vertex output variables. Any variable declared with the out qualifier (such as custom vertex attributes or the built-in gl_out array) becomes an array implicitly sized to N. While every invocation can read from any index of the input patch (gl_in[]) and even other indices in the output patch (provided barriers are used), an invocation is primarily responsible for writing data to its corresponding index, gl_out[gl_InvocationID].

Once all invocations finish and the tessellation levels (gl_TessLevelInner and gl_TessLevelOuter) are assigned, the resulting patch of \(N\) vertices is passed downstream to the Tessellation Evaluation Shader (TES). The TES then uses these \(N\) control points to calculate the final surface positions across the generated abstract tessellation domain.