Mesh Shader WorkGroupSize: Registers vs Occupancy

Tuning the gl_WorkGroupSize in GLSL mesh shaders is a critical optimization step that directly dictates how modern GPU compute units schedule work, allocate physical registers, and hide instruction latency. Selecting an optimal workgroup layout requires balancing General-Purpose Register (GPR) consumption per thread against maximum theoretical warp or wavefront occupancy. This article explores how workgroup dimensioning interacts with GPU hardware limits, how register pressure forces trade-offs with parallel thread execution, and practical strategies to determine the ideal workgroup size for mesh shading workloads.

The Mechanics of GPU Resource Allocation

Modern GPUs execute mesh shaders on hardware compute cores—such as NVIDIA Streaming Multiprocessors (SMs) or AMD Compute Units (CUs)—using SIMD execution units organized into warps (32 threads) or wavefronts (32 or 64 threads).

Each compute core has a finite, hard-partitioned register file (often 64 KB to 256 KB per core). When a mesh shader workgroup is dispatched, the hardware must allocate enough registers to satisfy the requirements of every thread within that workgroup simultaneously. If a workgroup cannot be granted its required registers and shared memory, it cannot be scheduled onto the core.

How gl_WorkGroupSize Affects Register Pressure

In GLSL, the layout qualifier sets the workgroup dimensions:

layout(local_size_x = 32, local_size_y = 1, local_size_z = 1) in;

The total thread count (\(X \times Y \times Z\)) directly scales the total register demand per workgroup.

  1. Smaller Workgroups (e.g., 32 to 64 threads):
    • Lower Aggregate Demand: A smaller workgroup consumes fewer total registers from the SM's physical register pool at any single moment.
    • Higher Per-Thread Register Ceiling: The compiler has more flexibility to allocate more registers per individual thread without causing register spilling to high-latency local scratch memory (VRAM).
    • Fine-Grained Scheduling: Smaller workgroups allow the scheduler to pack multiple groups into an SM to fit into leftover resource pockets.
  2. Larger Workgroups (e.g., 128 or 256 threads):
    • Bulk Register Footprint: A single workgroup claims a massive block of the register file.
    • Spill Risks: If the shader algorithm has high register pressure—due to complex transformations, lighting calculations, or unpacking intermediate geometry—the GPU may be forced to spill variables to VRAM, drastically reducing execution speed.

The Occupancy Trade-Off

Occupancy is the ratio of active warps/wavefronts on an SM relative to the theoretical maximum the hardware can support. High occupancy allows the GPU’s latency-hiding scheduler to switch to another warp whenever a thread stalls on memory reads or complex arithmetic.

Tuning gl_WorkGroupSize affects occupancy through two conflicting vectors:

  • Granularity Loss from Excessive Size: If an SM can support a maximum of 1,024 concurrent threads, setting gl_WorkGroupSize to 256 threads means a maximum of four workgroups can run concurrently. If register usage per thread slightly exceeds the hardware allocation bracket, the SM might only be able to host three workgroups (768 threads), dropping occupancy by 25%.
  • Wavefront Under-utilization from Mismatched Sizes: If gl_WorkGroupSize is not an integer multiple of the native SIMD size (e.g., setting a size of 48 on a 32-wide warp architecture), the hardware still executes full warps, leaving inactive lanes that waste register space and compute throughput.

Mesh Shader Specific Bottlenecks

Unlike standard compute shaders, mesh shaders output primitives and vertices through predefined output arrays:

layout(triangles, max_vertices = 64, max_primitives = 126) out;

This output architecture introduces unique pressure points:

  • I/O Storage vs. Registers: In some architectures, mesh shader output allocations reside in shared memory or dedicated primitive scratch spaces, which compete directly with the thread register budget for SM space.
  • Vertex-to-Thread Mapping: A 1:1 mapping between threads and output vertices often dictates the gl_WorkGroupSize. For instance, processing 64 vertices with 64 threads aligns cleanly with two 32-thread warps or one 64-thread wavefront.
  • Per-Primitive Divergence: When primitive generation requires dynamic culling (e.g., backface or frustum culling), having oversized workgroups can lead to severe SIMD branch divergence, wasting the allocated registers on idle threads.

Practical Tuning Strategies

To find the optimal balance between register consumption and occupancy:

  1. Align to Native SIMD Widths: Always configure total threads in multiples of 32 for NVIDIA (Warp32) or multiples of 32/64 for AMD (Wave32/Wave64). A common cross-vendor baseline is 32 or 64 threads.
  2. Profile with Platform Shader Analyzers: Use tools such as NVIDIA Nsight Graphics or AMD Radeon GPU Profiler (RGP) to inspect the exact register count (VGPRs/SGPRs) allocated per thread. Identify if reducing the workgroup size allows an additional workgroup to reside on the core.
  3. Prioritize Latency Hiding over 100% Occupancy: Aiming for 100% theoretical occupancy is often counterproductive if it limits the compiler to a low register budget, inducing spilling or register-reuse stalls. A moderate occupancy (around 50–70%) with zero spills and zero register-wait states typically yields the highest sustained instruction throughput.
  4. Decouple Threads from Primitives: If register pressure is high, reduce the workgroup size to 32 threads and use a processing loop where each thread computes multiple vertices or primitives sequentially, rather than scaling up the thread count.