Mesh Shader Register Pressure with PerprimitiveEXT

Declaring large perprimitiveEXT data structures in GLSL mesh shaders can cause significant register pressure, degrading GPU occupancy and rendering throughput. This article analyzes how hardware handles per-primitive outputs, how expansive data structures drive up General Purpose Register (GPR) usage, the resulting architectural penalties like thread spilling, and actionable practices to mitigate these performance bottlenecks.

Hardware Staging of Mesh Shader Outputs

Mesh shaders replace traditional vertex and primitive generation stages by operating across workgroups of threads that collaboratively output a meshlet (a collection of vertices and primitives). Outputs declared with perprimitiveEXT represent attributes associated with entire primitives (such as triangles or lines), rather than individual vertices.

Unlike standard vertex shader outputs that map cleanly to fixed-function interpolators, mesh shader outputs must be buffered on-chip before being handed off to the rasterizer. Hardware architectures typically stage these outputs in Workgroup Shared Memory (such as LDS on AMD or Shared Memory on NVIDIA) or dedicated hardware attribute buffers. However, individual threads responsible for calculating and writing these primitive attributes must first compute, manipulate, and hold the relevant values in thread-local registers (such as Vector General Purpose Registers, or VGPRs).

How Large Data Structures Drive Register Pressure

Declaring a large struct under perprimitiveEXT introduces several register-related complications:

  1. Extended Live Ranges: When a complex struct contains numerous vectors, matrices, or custom scalar fields, the compiler must retain the intermediate components in registers while calculations are completed. If calculations for multiple struct members are interleaved or depend on earlier vertex computations, the live range of those registers remains open across many cycles.
  2. Thread-to-Primitive Mismatches: Workgroups often contain more threads than primitives, or vice versa (for example, 32 threads emitting up to 64 primitives). In instances where a single thread writes multiple primitives in a loop or unrolled block, the compiler may attempt to precompute and hold several instances of the struct simultaneously in registers to maximize instruction-level parallelism (ILP).
  3. Synchronization Points: Mesh shaders often require an execution and memory barrier (barrier()) between vertex processing and primitive index generation. If primitive attributes are computed prior to the barrier but committed afterward, the values must reside in registers across the barrier, drastically extending register live ranges and preventing the compiler from recycling them.

Repercussions on GPU Occupancy and Spilling

The primary architectural consequence of high register usage is decreased wave/warp occupancy.

Every Streaming Multiprocessor (SM) or Compute Unit (CU) has a finite register file capacity. The hardware dynamically determines how many active warps or wavefronts can reside simultaneously on a compute unit based on the registers and shared memory demanded by each workgroup. If a single mesh shader thread requires 64 or 128 VGPRs instead of 32 due to large output staging:

  • Occupancy Drops: Fewer workgroups can execute concurrently on each core. This prevents the GPU from effectively hiding high-latency operations, such as texture fetches or global buffer reads.
  • Register Spilling to Scratch Memory: If the register requirement exceeds the maximum physical register allocation per thread, the compiler must spill live values into local scratch memory. Scratch memory is backed by off-chip VRAM with L1/L2 caching. Spilling converts fast register access into high-latency read/write memory transactions, causing execution stalls and instruction stalls.

Mitigation Strategies

To prevent perprimitiveEXT structures from inflating register footprints, use the following techniques:

  • Compute and Commit Immediately: Structure calculations so that a member of a perprimitiveEXT struct is written directly to the output array as soon as it is computed. Avoid staging the entire struct in a local variable before assigning it to the output.
  • Pack Attribute Formats: Replace standard 32-bit floats with 16-bit floats (float16_t) or packed normalized integer formats (e.g., uint containing four 8-bit channels via packUnorm4x8) for attributes like normals, tangent vectors, or material flags.
  • Defer to the Fragment Stage: Evaluate whether heavy per-primitive data can be derived analytically in the fragment shader or fetched directly from a storage buffer using the primitive ID (gl_PrimitiveID), completely removing the data payload from the mesh shader output interface.