layout(triangles) in GLSL Tessellation Shaders

Specifying layout(triangles) in a GLSL Tessellation Evaluation Shader (TES) defines the tessellation domain as a triangle, instructing the hardware tessellation primitive generator to subdivide the abstract patch into a mesh of smaller triangles. This layout configuration determines how the tessellation levels are interpreted, activates a three-component barycentric coordinate system via gl_TessCoord, and governs how the resulting vertices are connected to form the final primitive geometry.

Domain and Primitive Generation

When layout(triangles) is declared, the tessellation primitive generator operates on a triangular domain rather than a quad or an isoline. The generator subdivides this triangular area into smaller internal triangles based on the tessellation level factors provided by the Tessellation Control Shader (TCS) or the OpenGL API. The TES is then executed once for each vertex produced during this subdivision process.

Barycentric Coordinates (gl_TessCoord)

In a triangular domain, vertex positions are represented using barycentric coordinates rather than Cartesian coordinates. Within the shader, the built-in variable gl_TessCoord becomes a three-component vector (u, v, w) with the following properties:

• Each component ranges from 0.0 to 1.0.
• The sum of the three components always equals one: \(u + v + w = 1.0\).
• Each component represents the relative weight of one of the three corners of the patch triangle.

Shaders typically use these coordinates to interpolate per-vertex attributes across the patch using weighted sums, such as:

vec3 position = gl_TessCoord.x * p0 + 
                gl_TessCoord.y * p1 + 
                gl_TessCoord.z * p2;

Interpretation of Tessellation Levels

The layout(triangles) qualifier dictates which values of the inner and outer tessellation levels are consumed by the hardware primitive generator:

• Outer Tessellation Levels (gl_TessLevelOuter): Only the first three indices ([0], [1], and [2]) are used. Each value dictates how many segments the corresponding outer edge of the master triangle is divided into.
• Inner Tessellation Levels (gl_TessLevelInner): Only the first index ([0]) is used. This value controls the subdivision density of the interior region of the triangle. The second inner level index ([1]) is completely ignored.

If any of the active outer levels are less than or equal to zero, the patch is discarded entirely, and no vertices or primitives are generated for that patch.

Primitive Assembly

By default, the vertices computed by the TES are assembled into independent triangles following the winding order specified alongside the layout (such as cw for clockwise or ccw for counter-clockwise). If the point_mode layout qualifier is also included, the primitive generator instead outputs individual points at each evaluated vertex rather than connecting them into triangles.