Using gl_ClipDistance for User Clipping Planes in GLSL

This article explains how the built-in gl_ClipDistance array establishes custom user clipping planes within the OpenGL programmable pipeline. It covers the mathematical principles behind signed plane distances, the evaluation of clipping values in vertex and geometry shaders, the mandatory client-side OpenGL state setup, and the subsequent hardware interpolation and primitive discarding stages.

Overview of gl_ClipDistance

In GLSL, gl_ClipDistance is a built-in output float array available in vertex, tessellation, and geometry shader stages. Instead of relying purely on the fixed view-frustum clipping boundaries (near, far, left, right, top, bottom), developers use gl_ClipDistance to define arbitrary, user-specified clipping half-spaces. Each element of the array corresponds to an active clipping plane enabled in the OpenGL context.

The Mathematical Principle

A user clipping plane is defined mathematically in Hessian normal form by a four-component vector:

\[\mathbf{P} = (A, B, C, D)\]

Where \((A, B, C)\) represents the normalized normal vector of the plane, and \(D\) represents the distance offset from the origin. For any vertex position \(\mathbf{v} = (x, y, z, 1.0)\) in the same coordinate space (typically world space or view space), the signed distance \(d\) from the vertex to the plane is calculated using the dot product:

\[d = \mathbf{P} \cdot \mathbf{v} = Ax + By + Cz + D\]

  • If \(d \ge 0.0\), the vertex lies in the positive half-space (the visible region) and is preserved.
  • If \(d < 0.0\), the vertex lies in the negative half-space and is subject to clipping.

Assigning this scalar result to an index of gl_ClipDistance communicates the vertex's spatial relationship to that specific clipping plane.

OpenGL Host Configuration

Before the GPU hardware respects the values written to gl_ClipDistance, the corresponding clipping plane must be explicitly enabled through the OpenGL API on the host application side:

glEnable(GL_CLIP_DISTANCE0);
glEnable(GL_CLIP_DISTANCE1); // If using a second plane

If a plane index is written in GLSL but not enabled via glEnable(GL_CLIP_DISTANCEi), the hardware ignores the computed distance for that index. The maximum number of supported planes can be queried at runtime using glGetIntegerv(GL_MAX_CLIP_DISTANCES, &maxClipDistances), which is guaranteed to be at least 8 in modern implementations.

Implementation in the Vertex Shader

When no geometry or tessellation shader is present, distances are calculated directly in the vertex shader. The size of the gl_ClipDistance array must either be declared explicitly or implicitly sized through indexing.

#version 330 core

layout(location = 0) in vec3 aPos;

uniform mat4 uModel;
uniform mat4 uView;
uniform mat4 uProjection;
uniform vec4 uClipPlane; // Plane equation in world space (Ax + By + Cz + D)

out float gl_ClipDistance[1];

void main()
{
    vec4 worldPosition = uModel * vec4(aPos, 1.0);
    
    // Calculate signed distance to the plane
    gl_ClipDistance[0] = dot(worldPosition, uClipPlane);

    gl_Position = uProjection * uView * worldPosition;
}

The coordinate system used for the plane equation and vertex position must match. World space is preferred because it avoids recalculating plane equations when the camera moves.

Handling in the Geometry Shader

If a geometry shader is introduced into the pipeline, it intercepts the outputs of the vertex shader. The vertex shader's assignments to gl_ClipDistance do not automatically propagate to rasterization.

In the geometry shader:

  1. The incoming distances are read from gl_in[i].gl_ClipDistance[].
  2. The outgoing distances must be explicitly written to gl_ClipDistance[] for every generated vertex before calling EmitVertex().
#version 330 core

layout(triangles) in;
layout(triangle_strip, max_vertices = 3) out;

in float gl_ClipDistanceIn[][1]; // Custom in or via gl_in depending on GLSL version
out float gl_ClipDistance[1];

void main()
{
    for (int i = 0; i < 3; i++)
    {
        gl_Position = gl_in[i].gl_Position;
        // Pass through distance from vertex stage or calculate per-vertex
        gl_ClipDistance[0] = gl_in[i].gl_ClipDistance[0];
        EmitVertex();
    }
    EndPrimitive();
}

Hardware Interpolation and Primitive Clipping

Following the primitive assembly stage (post-vertex or post-geometry), the fixed-function rasterizer processes the gl_ClipDistance outputs:

  1. Interpolation: The scalar distance values are linearly interpolated across the faces of triangles and along lines.
  2. Boundary Determination: The hardware locates the zero-crossing points where interpolated distance equals \(0.0\).
  3. Primitive Reconstruction:
    • If all vertices of a primitive have values \(\ge 0.0\), the primitive passes unchanged.
    • If all vertices have values \(< 0.0\), the entire primitive is rejected before fragment processing.
    • If a primitive straddles the plane (having both positive and negative vertices), the hardware clips the geometry precisely along the plane boundary. New vertices are generated at the intersection line, interpolating all vertex attributes (UVs, normals, colors), and the exterior portion is discarded.