How gl_CullDistance Affects GLSL Rasterization

This article explains the mechanics of the gl_CullDistance built-in output in GLSL and its direct impact on primitive rasterization discard behavior. In OpenGL, gl_CullDistance provides hardware-accelerated, whole-primitive culling against user-defined half-spaces prior to rasterization. By writing signed floating-point distances to this array at the vertex, tessellation, or geometry shader stages, developers can cause entire primitives to be discarded before fragment shading occurs, bypassing the expensive vertex generation overhead associated with standard geometric clipping.

Understanding gl_CullDistance

The gl_CullDistance[] variable is a built-in shader output array of floating-point values available in pre-rasterization shader stages (Vertex, Tessellation Control, Tessellation Evaluation, and Geometry). Each element in the array represents the signed distance from the processed vertex to a user-defined plane or half-space:

  • Positive value or zero (>= 0.0): The vertex is inside the boundary (accepted).
  • Negative value (< 0.0): The vertex is outside the boundary (culled).

Unlike gl_ClipDistance, which clips geometry and computes new interpolated boundary vertices along the clipping boundary, gl_CullDistance determines visibility strictly at the primitive level.

Primitive Discard Rule

The OpenGL specification dictates that a primitive is completely discarded from rasterization if and only if all vertices forming that primitive have a negative cull distance for the same plane index.

Specifically, for an enabled cull plane index i:

  1. If vertex \(A\), vertex \(B\), and vertex \(C\) of a triangle all produce gl_CullDistance[i] < 0.0, the entire triangle is rejected and discarded before rasterization.
  2. If at least one vertex produces gl_CullDistance[i] >= 0.0, the primitive is retained and passes through to the rasterizer, even if the other two vertices have negative values.
  3. If multiple cull distances are enabled (e.g., indices 0 through n), the primitive is discarded if the all-negative condition is satisfied for any single enabled index.

Because interpolation does not create new vertices during a culling operation, partially culled primitives are rendered in their entirety unless paired with gl_ClipDistance or conventional viewport/frustum clipping.

Enabling Cull Planes on the Host

Setting values in gl_CullDistance within a shader has no effect on rasterization discard behavior unless the corresponding planes are enabled via the OpenGL API on the CPU side. Each cull distance index corresponds to a distinct capability:

glEnable(GL_CULL_DISTANCE0);
glEnable(GL_CULL_DISTANCE1);
// ... or using indexed enables where supported

If a shader writes to gl_CullDistance[i] but GL_CULL_DISTANCEi is not enabled, the hardware ignores the value for culling purposes, and the primitive is not discarded by that index.

Shader Implementation Example

In a vertex shader, gl_CullDistance is commonly calculated by taking the dot product of a vertex's position with a plane equation defined as \((A, B, C, D)\):

#version 450 core

layout(location = 0) in vec3 inPosition;

uniform mat4 uModelView;
uniform mat4 uProjection;
uniform vec4 uCullPlane; // Plane equation Ax + By + Cz + D = 0

out float gl_CullDistance[1];

void main()
{
    vec4 eyePos = uModelView * vec4(inPosition, 1.0);
    gl_Position = uProjection * eyePos;

    // Calculate signed distance to the plane
    gl_CullDistance[0] = dot(eyePos, uCullPlane);
}

If every vertex of a given primitive yields an eyePos that places it behind uCullPlane (resulting in a negative dot product), the hardware primitive assembler drops the primitive instantly.

Performance Impact on the Pipeline

Using gl_CullDistance to discard primitives provides significant performance benefits over manual fragment-stage discarding (discard in GLSL):

  • Bypasses Fragment Shading: Culled primitives are eliminated at the primitive assembly stage. No fragments are generated, saving execution time in the fragment shader and reducing bandwidth to the frame buffer.
  • Avoids Tessellation/Clipping Overhead: Unlike gl_ClipDistance, which splits triangles and produces new geometry that must be handled by the rasterizer, gl_CullDistance performs a binary keep-or-discard evaluation per primitive, preserving fixed-function pipeline throughput.