Geometry Shader Stalls with High Vertex Counts

Using GLSL geometry shaders to amplify primitives dynamically often leads to severe GPU pipeline stalls when configured to emit high vertex counts. This performance degradation is primarily driven by worst-case hardware memory allocations, thread execution divergence within SIMD units, backpressure on the primitive assembly pipeline, and the requirement to preserve primitive ordering. Understanding these architectural bottlenecks reveals why geometry shaders struggle with large expansion factors and how the underlying hardware handles variable-rate primitive generation.

Static On-Chip Memory Allocation and Occupancy Loss

The GPU hardware architecture requires fixed storage allocation for the output of each geometry shader thread. When you declare layout(max_vertices = N), the driver must reserve on-chip buffer space (such as local data share or specialized parameter caches) capable of holding the absolute worst-case scenario: \(N \times \text{size of output attributes}\).

This allocation happens statically per primitive invocation, regardless of whether a specific invocation emits \(N\) vertices or zero vertices. When max_vertices is large:

  • The required memory per thread group spikes dramatically.
  • Available on-chip registers and cache space are rapidly exhausted.
  • Occupancy plummets: The hardware scheduler can only launch a fraction of its normal concurrent waves or warps, leaving compute units underutilized and unable to hide memory latency.

SIMD Lane Divergence and Execution Imbalance

Modern GPUs process vertices and primitives in lockstep across SIMD vectors (typically 32 or 64 threads wide). In a geometry shader, vertex emission is controlled dynamically via runtime loops calling EmitVertex() and EndPrimitive().

When threads within the same wave produce varying numbers of vertices, execution diverges:

  1. Every thread in the SIMD wave must stay active until the thread with the highest emission count finishes.
  2. Lanes that finish early or emit fewer primitives are masked off and sit idle.
  3. This creates execution bubbles where hardware execution units run at a fraction of their capacity, stalling subsequent stages while waiting for the longest-running thread to complete.

Primitive Ordering and Assembly Backpressure

Graphics APIs strictly mandate that primitives arrive at the rasterizer in the exact sequence they were submitted by the draw call. Geometry shaders disrupt linear ordering by producing variable output streams:

  • The GPU’s Primitive Allocator and Geometry Assembler must reconstruct the original draw order from these dynamic, variable-length streams before passing them to the rasterizer.
  • If a geometry shader outputs a massive burst of vertices, the primitive export buffer fills rapidly.
  • Once the export queue is saturated, downstream hardware applies backpressure, physically stalling the geometry shader execution units until the rasterizer and fragment units consume the buffered primitives.

Memory Spilling to VRAM

When the declared max_vertices combined with the attribute payload exceeds the capacity of the GPU’s fast, on-chip geometry buffers, the driver is forced to spill intermediate primitive data to high-latency global device memory (VRAM). Round-tripping geometry through off-chip VRAM re-reads introduces high memory latency and bandwidth contention, directly causing pipeline stalls as shader cores wait for memory transfers to clear.

Architectural Solutions

Because of these fundamental hardware constraints, high-factor amplification in geometry shaders is inherently prone to stalls. Modern graphics pipelines avoid these issues by:

  • Using Compute Shaders with indirect draw calls to generate and write large amounts of geometry directly to buffers.
  • Implementing Mesh Shaders (via Task and Mesh stages), which replace the geometry shader with a cooperative thread-group model that allocates memory collectively rather than per-primitive, avoiding worst-case scalar allocation penalties.