Cull Patches with gl_TessLevelOuter in GLSL

In OpenGL and GLSL, the Tessellation Control Shader (TCS) provides a built-in, hardware-accelerated mechanism to discard entire patches before they reach the Tessellation Evaluation Shader (TES) or the rasterizer. By assigning a value of zero or less to any component of gl_TessLevelOuter[], the primitive generator completely skips generating geometry for that patch. This article details the underlying rules of this culling behavior, provides a practical GLSL implementation, and highlights key optimization use cases.

The Tessellation Culling Mechanism

The tessellation pipeline consists of three stages: the Tessellation Control Shader (TCS), the fixed-function Tessellation Primitive Generator (TPG), and the Tessellation Evaluation Shader (TES).

The TPG determines how many primitives to produce based on the values in gl_TessLevelOuter and gl_TessLevelInner. According to the OpenGL and GLSL specifications, if any relevant component of gl_TessLevelOuter is less than or equal to zero (or is a NaN), the primitive generator will produce zero primitives for that patch. Because no primitives are emitted, the TES is never executed for the discarded patch, saving significant vertex, geometry, and fragment processing cycles downstream.

Per-Patch Rules by Primitive Type

The specific element of gl_TessLevelOuter that triggers culling depends on the patch type configured in the evaluation shader:

  • Triangles (layout(triangles) in;): Evaluates gl_TessLevelOuter[0], gl_TessLevelOuter[1], and gl_TessLevelOuter[2]. If any of these three values is \(\le 0.0\), the patch is culled.
  • Quads (layout(quads) in;): Evaluates gl_TessLevelOuter[0] through gl_TessLevelOuter[3]. If any of these four values is \(\le 0.0\), the patch is culled.
  • Isolines (layout(isolines) in;): Evaluates gl_TessLevelOuter[0] and gl_TessLevelOuter[1]. If gl_TessLevelOuter[0] or gl_TessLevelOuter[1] is \(\le 0.0\), no lines are generated.

GLSL Implementation Example

Tessellation control shaders run once per output control point. Because gl_TessLevelOuter is a per-patch attribute, it is standard practice to compute visibility and write the tessellation levels exclusively on invocation zero (gl_InvocationID == 0) to prevent redundant calculations and race conditions.

#version 450 core

layout(vertices = 3) out;

in vec3 vPosition[];
out vec3 tcPosition[];

// Custom function to determine if the patch is visible
bool isPatchVisible() {
    // Example: Test patch bounding sphere against the view frustum
    // Returns false if the patch is outside the view frustum
    return true; 
}

void main() {
    // Pass through per-vertex attributes
    tcPosition[gl_InvocationID] = vPosition[gl_InvocationID];

    // Compute patch-level properties only once per patch
    if (gl_InvocationID == 0) {
        if (!isPatchVisible()) {
            // Setting any outer level to 0.0 culls the entire patch
            gl_TessLevelOuter[0] = 0.0;
            gl_TessLevelOuter[1] = 0.0;
            gl_TessLevelOuter[2] = 0.0;
            gl_TessLevelInner[0] = 0.0;
        } else {
            // Set normal tessellation factors for visible geometry
            gl_TessLevelOuter[0] = 4.0;
            gl_TessLevelOuter[1] = 4.0;
            gl_TessLevelOuter[2] = 4.0;
            gl_TessLevelInner[0] = 4.0;
        }
    }
}

Common Culling Applications

Setting outer levels to zero in the TCS is widely used for:

  1. View Frustum Culling: Computing a simple axis-aligned bounding box (AABB) or bounding sphere from the input control points in clip space. If the bounds fall outside the clip volume, writing zero to the outer levels drops the patch immediately.
  2. Back-Face Culling: Computing the patch normal from the control points. If the surface normal faces away from the camera, the patch can be discarded. Caution must be taken if substantial displacement mapping is applied in the TES, as displacement can shift back-facing surfaces into view.
  3. Sub-Pixel / Screen-Size Culling: If a patch projected onto screen coordinates occupies an area smaller than a pixel or a user-defined threshold, it can be dropped entirely to avoid rendering imperceptible geometry.