How Does TCS Set Subdivision Levels in GLSL?

Tessellation Control Shaders (TCS) determine the subdivision density of geometric primitives in OpenGL by computing and assigning tessellation level factors to built-in output arrays. This article examines how the TCS operates within the modern OpenGL pipeline, breaks down the roles of the outer and inner tessellation levels across different patch topologies, and demonstrates the mathematical strategies used to dynamically compute level-of-detail (LOD) based on distance, screen coverage, and edge length.

The Role of the Tessellation Control Shader

In the OpenGL rendering pipeline, tessellation sits between the vertex shader and the geometry or fragment shader. The tessellation stage consists of three components: the Tessellation Control Shader (TCS), the fixed-function Tessellation Primitive Generator (TPG), and the Tessellation Evaluation Shader (TES).

The primary responsibility of the TCS is twofold:

  1. Transforming or passing through control points that define a patch primitive.
  2. Determining how finely the fixed-function TPG should subdivide the input patch.

The subdivision density is communicated directly to the hardware generator using two built-in float arrays: gl_TessLevelOuter and gl_TessLevelInner.

Understanding Built-in Tessellation Levels

The fixed-function stage reads the values written to gl_TessLevelOuter and gl_TessLevelInner by the TCS invocation. The dimensions and meanings of these arrays depend entirely on the patch domain defined in the pipeline.

Triangle Domains (layout(triangles))

For triangular patches, three outer edges and one interior area require subdivision parameters:

Quad Domains (layout(quads))

Quad patches require four outer edge factors and two directional inner factors:

Isoline Domains (layout(isolines))

For isoline rendering:

If any outer level is set to zero or less, the primitive generator culls the entire patch, which serves as an effective mechanism for primitive-level frustum culling within the TCS.

Common Strategies for Calculating Subdivision Levels

Rather than hardcoding static values, graphics applications dynamically compute tessellation levels based on runtime parameters.

1. Distance-Based Level of Detail (LOD)

A standard approach calculates the Euclidean distance from the camera to the midpoint of each patch edge. Edges closer to the viewer receive higher subdivision values, while distant edges receive lower values:

float calculateDistanceLevel(vec3 p0, vec3 p1, vec3 cameraPos) {
    vec3 midPoint = (p0 + p1) * 0.5;
    float dist = distance(midPoint, cameraPos);
    return clamp(100.0 / dist, 1.0, 64.0);
}

2. Screen-Space Edge Length

To ensure visual consistency across various screen resolutions, edge subdivisions can be computed by projecting patch vertices into screen space. The screen-space distance in pixels between adjacent vertices dictates the tessellation factor, maintaining roughly equal triangle sizes on screen:

float screenSpaceLevel(vec4 clip0, vec4 clip1, vec2 viewportSize) {
    vec2 ndc0 = clip0.xy / clip0.w;
    vec2 ndc1 = clip1.xy / clip1.w;
    vec2 screen0 = (ndc0 * 0.5 + 0.5) * viewportSize;
    vec2 screen1 = (ndc1 * 0.5 + 0.5) * viewportSize;
    float pixelLength = distance(screen0, screen1);
    return clamp(pixelLength / 16.0, 1.0, 64.0);
}

3. Maintaining Edge Continuity

To prevent visual cracks and stitching artifacts between adjacent patches, shared edges must compute identical outer tessellation levels. Calculating gl_TessLevelOuter strictly based on the two endpoints of that specific edge ensures that neighboring patches compute the exact same tessellation factor for their shared boundary.

GLSL Implementation Example

The following TCS example takes a 3-vertex triangle patch, executes computations once per patch using invocation ID zero, and computes distance-based outer and inner subdivision levels:

#version 450 core

layout(vertices = 3) out;

in vec3 vPosition[];
out vec3 tcPosition[];

uniform vec3 uCameraPosition;

void main() {
    tcPosition[gl_InvocationID] = vPosition[gl_InvocationID];

    if (gl_InvocationID == 0) {
        float edge0 = clamp(100.0 / distance((vPosition[1] + vPosition[2]) * 0.5, uCameraPosition), 1.0, 32.0);
        float edge1 = clamp(100.0 / distance((vPosition[2] + vPosition[0]) * 0.5, uCameraPosition), 1.0, 32.0);
        float edge2 = clamp(100.0 / distance((vPosition[0] + vPosition[1]) * 0.5, uCameraPosition), 1.0, 32.0);

        gl_TessLevelOuter[0] = edge0;
        gl_TessLevelOuter[1] = edge1;
        gl_TessLevelOuter[2] = edge2;

        gl_TessLevelInner[0] = (edge0 + edge1 + edge2) / 3.0;
    }
}

By assigning programmatic values to gl_TessLevelOuter and gl_TessLevelInner, the Tessellation Control Shader dynamically directs the hardware tessellator to scale geometric complexity to match runtime visual requirements.