Understanding layout(isolines) in GLSL Shaders

The layout(isolines) execution mode in GLSL Tessellation Evaluation Shaders (TES) directs the hardware tessellator to generate sets of independent line strips across a normalized two-dimensional parameter space. Instead of generating triangles or quads, the tessellation primitive generator evaluates the outer tessellation levels to determine both the total number of parallel lines and the degree of subdivision along each line. This configuration is widely used for rendering hair, curves, contours, and wireframe grid effects.

Domain Definition and Coordinates

When layout(isolines) is specified, the tessellation domain is defined as a unit square \([0, 1] \times [0, 1]\). Inside the Tessellation Evaluation Shader, the input variable gl_TessCoord provides the barycentric or parametric coordinate of the generated vertex:

  • gl_TessCoord.x represents the position along an individual isoline strip, ranging from \(0.0\) to \(1.0\).
  • gl_TessCoord.y represents the normalized offset of the current line strip within the domain.
  • gl_TessCoord.z is unused and set to \(0.0\).

Each line strip is generated parallel to the u-axis (\(x\)), holding the v-axis (\(y\)) constant across the entire length of that strip.

Tessellation Levels

The isoline generator evaluates only the first two entries of gl_TessLevelOuter. The remaining outer levels and all inner levels (gl_TessLevelInner) are ignored:

  • gl_TessLevelOuter[0] (Line Count): Determines the number of isolines to generate. The value is clamped to implementation limits and rounded to an integer. If this level is less than or equal to zero, all geometry is culled. Unlike standard quad or triangle tessellation, gl_TessLevelOuter[0] produces \(n\) lines distributed evenly across the domain, typically at \(y = \frac{i}{n}\) where \(i \in [0, n-1]\).
  • gl_TessLevelOuter[1] (Segment Count): Determines how many segments each line strip is divided into. A value of \(m\) segments produces \(m + 1\) vertices per line strip along the x-axis.

Spacing and Subdivision Behavior

Subdivision modes defined in the TES layout declaration interact with isolines in specific ways:

  • equal_spacing: The segment count defined by gl_TessLevelOuter[1] is rounded to the nearest integer, and segments are partitioned equally across \([0, 1]\).
  • fractional_even_spacing and fractional_odd_spacing: These modes apply exclusively to gl_TessLevelOuter[1]. Vertices along the strip adjust smoothly as the tessellation level changes dynamically, preventing sudden popping artifacts along the curve.
  • Line Count Spacing: gl_TessLevelOuter[0] is not affected by fractional spacing modes; it always behaves as an integer division without smooth blending between line counts.

Primitive Output

The output of the tessellator in this mode is a sequence of non-connecting line strips. Each strip contains vertices ordered sequentially from gl_TessCoord.x = 0.0 to gl_TessCoord.x = 1.0. The primitives are emitted directly to downstream pipeline stages (such as the Geometry Shader or Rasterizer) as individual line segments within each strip. If the optional point_mode layout qualifier is enabled, the hardware skips line strip assembly and renders every evaluated vertex as an individual point primitive instead.