Mesh Shader Output Primitive Limits in GLSL

Mesh shaders streamline the graphics pipeline by giving developers compute-like control over geometry generation, but the maximum number of output primitives a single workgroup can emit is bounded by physical hardware limits. This article examines the core hardware and architectural constraints—such as on-chip cache memory, vertex and primitive attribute allocations, register pressure, and rasterizer interface bottlenecks—that dictate the upper bounds of maxMeshOutputPrimitives in GLSL mesh shaders.

Specification Caps and Implementation Limits

In GLSL (via GL_EXT_mesh_shader or VK_EXT_mesh_shader), the maximum output primitive count for a workgroup is constrained upfront by the implementation limit maxMeshOutputPrimitives. While the Vulkan specification guarantees a minimum value (typically 256 primitives), the absolute ceiling is determined by the physical GPU architecture. On current desktop hardware (such as NVIDIA Turing/Ampere/Ada Lovelace and AMD RDNA 2/RDNA 3), this limit is commonly fixed at 256 or 512 primitives per workgroup.

On-Chip Primitive and Vertex Storage (SRAM / LDS)

The most direct physical constraint is the size of the on-chip memory allocated to store workgroup outputs before they are passed to the rasterizer. When a mesh shader executes, it must allocate space for:

  1. Per-Primitive Indices: Each triangle primitive requires a 3-component index tuple (typically 8-bit or 16-bit integers). A limit of 256 triangles requires dedicated storage just to track connectivity.
  2. Per-Vertex Attributes: Positions, normals, texture coordinates, and custom user data written by the workgroup.
  3. Per-Primitive Attributes: Primitive flags, layer selections, or custom primitive attributes.

On architectures like AMD RDNA, this data resides in the Local Data Store (LDS); on NVIDIA, it is held in a dedicated output partition of the shared memory / register cache. Because this high-speed SRAM is strictly limited per Streaming Multiprocessor (SM) or Compute Unit (CU), allocating excessive primitive outputs would consume the entire local storage, preventing other wave/warp slots from executing concurrently.

Register Pressure and Thread Allocation

Mesh shaders decouple the thread count from the vertex and primitive count. For example, a workgroup of 32 or 64 threads may emit 128 or 256 primitives.

To track this dynamic generation, the hardware maintains tracking metadata per thread. Higher primitive counts require more loop iterations or more variables maintained in general-purpose registers (VGPRs). As register allocation per thread increases, thread occupancy drops. Hardware architects place a hard limit on total output primitives to guarantee that the hardware's internal bookkeeping tables do not exceed hardware register allocations.

Fixed-Function Primitive Allocator and Rasterizer Bandwidth

Once a mesh shader workgroup completes, its output is handed off to the primitive assembler and the hardware rasterizer. Hardware primitive engines have fixed-width input buffers that can only ingest a specific number of primitives per clock cycle.

If a single workgroup were permitted to output thousands of primitives:

  • The internal primitive FIFO queues between the compute units and the rasterizer would easily overflow.
  • Workgroups would stall waiting for rasterizer lanes to free up, causing pipeline bubbles.
  • Dynamic vertex reuse efficiency would degrade, as the cache line sizes for primitive distribution are tuned for clusters of around 64 to 128 vertices and up to 256 primitives.

Optimal Balance

The hardware limit on output primitives is not merely a theoretical ceiling, but a balance point between parallel utilization and physical chip layout. Capping primitive generation per workgroup (commonly to 256 triangles) guarantees that geometry fits inside fast L1-adjacent storage, maintains balanced SIMD occupancy, and matches the processing ingestion rates of downstream raster hardware.