How Do GLSL Geometry Shaders Generate Primitives?

Geometry shaders occupy a unique position in the OpenGL rendering pipeline, sitting between the vertex shader and rasterization stage with the ability to create or destroy geometry on the GPU. Unlike vertex shaders, which operate on single vertices in a strict 1:1 input-to-output mapping, geometry shaders receive entire base primitives and dynamically generate new vertices and primitives using built-in emission functions. This article explores how geometry shaders declare primitive layouts, control dynamic output amplification using EmitVertex() and EndPrimitive(), and manage GPU memory constraints.

The Role of the Geometry Shader in the Pipeline

In a standard graphics pipeline, the vertex shader processes incoming attributes per vertex. The primitive assembly stage groups these vertices into primitives such as points, lines, or triangles.

When enabled, the geometry shader receives these assembled primitives as input arrays. Because it processes complete primitives rather than isolated vertices, it can inspect adjacent topological data, calculate face normals, subdivide surfaces, or discard primitives entirely before the rasterizer converts them into fragments.

Layout Qualifiers: Defining Inputs and Outputs

Every GLSL geometry shader requires explicit layout declarations to inform the GPU driver of the input primitive type, the output primitive type, and the maximum number of vertices the shader can produce in a single execution.

Input Layouts

The input layout defines the primitive type accepted from the vertex assembly stage. Common input types include:

• points
• lines
• lines_adjacency
• triangles
• triangles_adjacency

#version 330 core
layout (points) in;

When receiving points, the built-in gl_in[] array contains 1 element. When receiving triangles, gl_in[] contains 3 elements corresponding to the vertices of the triangle.

Output Layouts and Memory Limits

The output layout defines the geometry format emitted to the rasterizer, along with a strict upper bound on emitted vertices:

• points
• line_strip
• triangle_strip

layout (triangle_strip, max_vertices = 4) out;

The max_vertices qualifier is critical for hardware scheduling. The GPU allocates a fixed output buffer per invocation based on this value. Setting max_vertices excessively high can decrease thread occupancy and degrade performance, even if the shader emits fewer vertices at runtime.

Generating Primitives with EmitVertex() and EndPrimitive()

Dynamic primitive generation in GLSL is driven by two intrinsic functions: EmitVertex() and EndPrimitive().

EmitVertex()

Calling EmitVertex() tells the GPU to capture all current values assigned to output variables (such as gl_Position, colors, or texture coordinates) and record them as a single completed vertex. Once emitted, the shader can modify those variables to define the next vertex.

EndPrimitive()

Calling EndPrimitive() finalizes the current strip (either line_strip or triangle_strip). Subsequent calls to EmitVertex() will begin constructing a new, disconnected primitive strip within the same shader invocation.

If a shader finishes execution without calling EndPrimitive(), the GPU automatically finalizes whatever strip is currently open. If no vertices were emitted during the execution, the incoming primitive is effectively culled.

Practical Example: Expanding Points into Quads

A common application of geometry shaders is camera-facing billboard generation, where a single input point is expanded into a four-vertex quad (two connected triangles) on the fly.

#version 330 core
layout (points) in;
layout (triangle_strip, max_vertices = 4) out;

out vec2 TexCoords;

uniform float u_Size;

void main() {
    vec4 center = gl_in[0].gl_Position;

    // Bottom-Left
    gl_Position = center + vec4(-u_Size, -u_Size, 0.0, 0.0);
    TexCoords = vec2(0.0, 0.0);
    EmitVertex();

    // Bottom-Right
    gl_Position = center + vec4(u_Size, -u_Size, 0.0, 0.0);
    TexCoords = vec2(1.0, 0.0);
    EmitVertex();

    // Top-Left
    gl_Position = center + vec4(-u_Size, u_Size, 0.0, 0.0);
    TexCoords = vec2(0.0, 1.0);
    EmitVertex();

    // Top-Right
    gl_Position = center + vec4(u_Size, u_Size, 0.0, 0.0);
    TexCoords = vec2(1.0, 1.0);
    EmitVertex();

    EndPrimitive();
}

In this implementation:

1. The CPU issues a draw call sending only point primitives.
2. The vertex shader transforms each point.
3. The geometry shader executes once per point, emitting four offset vertices.
4. Because the output layout is triangle_strip, four sequential vertices define two adjacent triangles forming a quad.

Common Use Cases

• Particle Systems and Billboards: Generating screenspace or worldspace billboards from 1-vertex points minimizes CPU-to-GPU bandwidth.
• Wireframe Overlays: Geometry shaders with triangles input can calculate barycentric coordinates per triangle, allowing the fragment shader to render sharp wireframe lines without duplicate meshes.
• Extrusion and Fur Effects: Geometry shaders can project vertices outward along surface normals to generate extruded silhouettes, shadow volume fins, or layered fur.
• Layered Rendering: By outputting to the gl_Layer built-in variable, a geometry shader can duplicate a single mesh into multiple cube map faces or texture array slices in a single pass (often used for point light shadow maps).

Performance Considerations

While geometry shaders provide flexible dynamic geometry generation, they carry performance trade-offs on modern hardware architectures:

• Variable Output Rates: GPUs excel at fixed-function workloads. The variable output size per thread in a geometry shader can cause thread divergence and pipeline bubbles.
• Memory Stalls: Setting a large max_vertices value forces the GPU to reserve significant on-chip register space, reducing the number of concurrent warps or wavefronts.
• Modern Alternatives: For heavy geometric amplification, modern APIs and extensions favor compute shaders with indirect draw calls or mesh shading pipelines (task and mesh shaders), which offer more predictable parallel scheduling and greater memory control.