How Does the OpenGL Tessellation Evaluation Shader Work?

The Tessellation Evaluation Shader (TES) is a programmable stage in modern OpenGL that calculates the final vertex attributes for primitives generated by the tessellation pipeline. Positioned directly after the fixed-function primitive generator, the TES executes once for every newly generated vertex within an abstract patch. This article examines how the TES fits into the pipeline, explores its input parameters—such as barycentric and normalized domain coordinates—and explains how it applies displacement maps, surface equations, and coordinate transformations before forwarding geometric data to downstream stages.

Positioning in the OpenGL Rendering Pipeline

The tessellation subsystem in OpenGL consists of three sequential components that sit between the Vertex Shader and the Geometry (or Fragment) Shader:

1. Tessellation Control Shader (TCS): Determines the inner and outer tessellation levels (gl_TessLevelInner and gl_TessLevelOuter) and processes per-patch input control points into output control points.
2. Primitive Generator (PG): A fixed-function hardware stage that subdivides an abstract domain (quad, triangle, or isoline) according to the tessellation levels specified by the TCS. It generates a grid or mesh of vertices, assigning each a parametric coordinate without any knowledge of 3D world space.
3. Tessellation Evaluation Shader (TES): A programmable stage that takes the abstract coordinates produced by the primitive generator, samples the patch control points, and produces actual geometric vertices in clip space, view space, or world space.

While the TCS defines how finely a surface is divided, the TES dictates where those subdivisions actually exist in three-dimensional space.

Domain Types and Parametric Coordinates

The TES begins with layout qualifiers that define how it interprets the abstract coordinates provided by the primitive generator:

layout(triangles, equal_spacing, ccw) in;

• Domain Geometry: OpenGL supports three domain types: quads, triangles, and isolines.
• Spacing Modes: Subdivisions can be configured as equal_spacing, fractional_even_spacing, or fractional_odd_spacing, which govern how new vertices appear smoothly as tessellation levels change dynamically.
• Primitive Orientation: The vertex winding order can be explicitly declared as counter-clockwise (ccw) or clockwise (cw).

The primary input consumed by the TES is the built-in variable gl_TessCoord, which represents the parametric location of the current vertex within the patch:

• Triangular Domains: gl_TessCoord contains three-component barycentric coordinates \((u, v, w)\), where \(u + v + w = 1.0\). Each component corresponds to the weight of a control point forming the triangular patch.
• Quad and Isoline Domains: gl_TessCoord functions as a two-dimensional Cartesian coordinate \((u, v)\), spanning the range \([0.0, 1.0]\) across both axes.

Evaluating Positions and Attributes

Because the fixed-function primitive generator only outputs dimensionless coordinates, the TES must interpolate user data across the patch control points.

For a triangular domain, barycentric interpolation combines attributes from the patch's vertices:

vec3 p0 = gl_in[0].gl_Position.xyz;
vec3 p1 = gl_in[1].gl_Position.xyz;
vec3 p2 = gl_in[2].gl_Position.xyz;

vec3 position = gl_TessCoord.x * p0 +
                gl_TessCoord.y * p1 +
                gl_TessCoord.z * p2;

For quad domains, bilinear or bicubic evaluation (such as Bézier surface patches) is used:

vec3 p0 = mix(gl_in[0].gl_Position.xyz, gl_in[1].gl_Position.xyz, gl_TessCoord.x);
vec3 p1 = mix(gl_in[2].gl_Position.xyz, gl_in[3].gl_Position.xyz, gl_TessCoord.x);
vec3 position = mix(p0, p1, gl_TessCoord.y);

In addition to position, any vertex attribute—such as texture coordinates, tangent vectors, and normals—must be explicitly interpolated in the TES if it is needed by downstream shaders.

Surface Displacement and Transformation

A primary use case for the TES is hardware-accelerated displacement mapping and procedural deformation. Because new geometry has been instantiated at this stage, high-frequency surface detail can be sampled directly from texture maps.

After computing the base surface position and surface normal via interpolation, the TES can sample a heightmap:

float displacement = texture(heightMap, interpolatedTexCoord).r;
position += normal * (displacement * displacementScale);

gl_Position = projectionMatrix * viewMatrix * vec4(position, 1.0);

By pushing geometric vertices along the normal vector inside the TES, rendering engines generate genuine silhouette detail and depth without requiring high-density meshes to be transferred across the CPU-GPU bus.

Downstream Pipeline Hand-off

The output of the TES behaves identically to the output of a standard Vertex Shader. The shader writes to the built-in output variable gl_Position and forwards any custom output variables (declared with the out qualifier).

Once execution finishes for all generated vertices in a patch, the primitives are assembled into geometric triangles or lines. They either pass into an optional Geometry Shader, feed into the transform feedback buffer, or proceed directly to the rasterizer for interpolation across fragments.