GLSL Mesh Shader Output Vertex and Primitive Limits

This article provides a concise overview of the vertex and primitive output limits enforced on mesh shader workgroups in GLSL under cross-vendor and vendor-specific specifications. Mesh shaders replace traditional fixed-function geometry pipelines by operating on flexible, compute-like workgroups. To ensure optimal memory usage and hardware scheduling, graphics APIs place hard limitations and recommended thresholds on the maximum number of vertices and primitives a single workgroup can generate.

API Specification Minimum Limits

Under the standard cross-vendor extension (GL_EXT_mesh_shader / VK_EXT_mesh_shader), the specification establishes guaranteed minimum capabilities that all conforming hardware must support:

  • Maximum Output Vertices (maxMeshOutputVertices): 256
  • Maximum Output Primitives (maxMeshOutputPrimitives): 256

While an implementation can expose limits higher than 256, current GPU architectures across NVIDIA (Turing, Ampere, Ada Lovelace), AMD (RDNA 2, RDNA 3), and Intel (Arc) universally adhere to a maximum limit of 256 vertices and 256 primitives per workgroup.

GLSL Declaration Constraints

In GLSL, the output limits must be declared statically at compile time using layout qualifiers within the mesh shader:

#version 460
#extension GL_EXT_mesh_shader : require

layout(local_size_x = 32, local_size_y = 1, local_size_z = 1) in;
layout(triangles, max_vertices = 256, max_primitives = 256) out;

Compile-time validation ensures that:

  1. max_vertices cannot exceed the implementation limit gl_MaxMeshOutputVerticesNV or the queried Vulkan device property maxMeshOutputVertices.
  2. max_primitives cannot exceed gl_MaxMeshOutputPrimitivesNV or the queried maxMeshOutputPrimitives.

At runtime, the shader dynamically sets the actual count for that invocation using SetMeshOutputsEXT(vertex_count, primitive_count). The runtime values cannot exceed the compile-time limits declared in the layout qualifiers.

Memory Limits and Sizing Considerations

In addition to count limits, outputs are constrained by physical hardware memory allocations per workgroup:

  • Maximum Mesh Output Memory (maxMeshOutputMemorySize): Standard specifications mandate at least 32,768 bytes (32 KB).

Total output memory is calculated by summing:

  1. Per-vertex attributes: max_vertices * sizeof(all per-vertex output attributes)
  2. Per-primitive attributes: max_primitives * sizeof(all per-primitive output attributes)
  3. Primitive indices: max_primitives * index_count * sizeof(uint)

If the declared vertex and primitive limits cause the combined payload size to exceed maxMeshOutputMemorySize, the shader will fail to compile or link.

Optimal Workgroup Sizing

Although hardware supports up to 256 vertices and 256 primitives, allocating the maximum limits can reduce thread occupancy by consuming substantial register and on-chip shared memory space. For real-world triangle meshlets, developers typically target smaller layouts—such as 64 vertices and 126 primitives—to maximize cache efficiency, minimize redundant vertex processing, and maintain high wave/warp occupancy on the GPU.