Understanding gl_TessLevelOuter in GLSL Tessellation

This article provides an overview of how to configure the gl_TessLevelOuter array in GLSL Tessellation Control Shaders (TCS) to control perimeter subdivision. It details the specific floating-point values required for different primitive domains—triangles, quads, and isolines—explaining how each index corresponds to specific patch edges and how these values dictate mesh density and geometry culling.

Overview of gl_TessLevelOuter

In the Tessellation Control Shader, gl_TessLevelOuter is a built-in array of four floating-point numbers (float gl_TessLevelOuter[4]). These values specify how many segments the outer boundary edges of an abstract patch domain should be divided into by the fixed-function Tessellation Primitive Generator (TPG).

General rules governing these values include:

  • Value Type: Floating-point numbers. Depending on the spacing mode defined in the Tessellation Evaluation Shader (TES)—such as equal_spacing, fractional_even_spacing, or fractional_odd_spacing—floating-point values are rounded or smoothly interpolated to partition edges.
  • Culling: Setting any relevant gl_TessLevelOuter value to 0.0 or less (or NaN) results in the entire patch being culled and discarded before rasterization.
  • Hardware Limits: Values are clamped between 1.0 and the implementation-dependent maximum limit, queried via GL_MAX_TESS_GEN_LEVEL (commonly 64).

Triangle Domains (triangles)

When tessellating triangles, only the first three elements of gl_TessLevelOuter are used. Each element dictates the subdivision factor for the perimeter edge opposite the vertex of the same index:

  • gl_TessLevelOuter[0]: Controls the subdivision of the edge spanning from vertex 1 to vertex 2 (the edge opposite vertex 0).
  • gl_TessLevelOuter[1]: Controls the subdivision of the edge spanning from vertex 2 to vertex 0 (the edge opposite vertex 1).
  • gl_TessLevelOuter[2]: Controls the subdivision of the edge spanning from vertex 0 to vertex 1 (the edge opposite vertex 2).

gl_TessLevelOuter[3] is ignored for triangular patches.

Quad Domains (quads)

For quadrilateral patches, all four elements of gl_TessLevelOuter are utilized. They correspond to the four normalized parameter edges of the \([0, 1] \times [0, 1]\) domain coordinate space:

  • gl_TessLevelOuter[0]: Left edge, where the horizontal coordinate \(u = 0\) (spans along the vertical axis from \(v = 0\) to \(v = 1\)).
  • gl_TessLevelOuter[1]: Bottom edge, where the vertical coordinate \(v = 0\) (spans along the horizontal axis from \(u = 0\) to \(u = 1\)).
  • gl_TessLevelOuter[2]: Right edge, where the horizontal coordinate \(u = 1\) (spans along the vertical axis from \(v = 0\) to \(v = 1\)).
  • gl_TessLevelOuter[3]: Top edge, where the vertical coordinate \(v = 1\) (spans along the horizontal axis from \(u = 0\) to \(u = 1\)).

Isoline Domains (isolines)

Isoline tessellation decomposes the domain into a series of separate line segments. Only the first two array entries are evaluated:

  • gl_TessLevelOuter[0]: Defines the number of distinct lines generated across the patch. This value is clamped and rounded to an integer.
  • gl_TessLevelOuter[1]: Defines the number of segments that comprise each individual line, dictating line detail.

gl_TessLevelOuter[2] and gl_TessLevelOuter[3] are ignored for isoline patches.