Using SetMeshOutputsEXT in GLSL Mesh Shaders

This article explains how the SetMeshOutputsEXT built-in function operates in GLSL mesh shaders under the VK_EXT_mesh_shader extension. It details how the function dynamically sets runtime vertex and primitive counts, how it interacts with compile-time maximum allocation limits, the execution rules required to use it safely, and a concise code example demonstrating its proper integration.

Compile-Time Maximums vs. Runtime Allocations

When writing a mesh shader in GLSL, you must declare the maximum number of vertices and primitives a workgroup can ever emit using compile-time layout qualifiers:

layout(local_size_x = 32) in;
layout(triangles, max_vertices = 64, max_primitives = 126) out;

These qualifiers define the upper memory bounds the hardware allocates for the workgroup. However, a workgroup rarely needs to output the maximum capacity every time, especially when performing GPU-driven culling (such as back-face or frustum culling). SetMeshOutputsEXT provides the mechanism to declare the exact number of vertices and primitives generated during a specific workgroup execution.

How SetMeshOutputsEXT Functions

The function signature is defined as:

void SetMeshOutputsEXT(uint vertexCount, uint primitiveCount);

When called, it informs the rasterizer and downstream pipeline stages of the actual topology size:

  • vertexCount: The dynamic number of vertices this workgroup will populate in output arrays like gl_MeshVerticesEXT[].
  • primitiveCount: The dynamic number of primitives (e.g., triangles, lines, or points) this workgroup will define in index arrays like gl_PrimitiveTriangleIndicesEXT[].

Both parameters must be less than or equal to the values declared in max_vertices and max_primitives. Passing values greater than the declared maximums results in undefined behavior. If either count is set to 0, the pipeline emits no geometry for that category, allowing workgroups to discard entire meshlets with zero primitive generation overhead.

Execution Rules and Placement

To use SetMeshOutputsEXT correctly, you must adhere to several execution constraints:

  1. Order of Execution: You must call SetMeshOutputsEXT before writing to any per-vertex or per-primitive output arrays. Writing to outputs before setting their counts is invalid.
  2. Workgroup Uniformity: The function modifies the output state for the entire workgroup. While it does not strictly need to be called by every thread, if multiple threads call it, they must pass identical values. The standard practice is to let invocation 0 set the counts or compute the counts uniformly across the workgroup.
  3. Single Execution: The function should be executed once per workgroup before the output phase. Calling it multiple times with varying values within the same workgroup leads to undefined behavior.

Basic Implementation Example

The following pattern demonstrates how dynamic counts are evaluated and assigned before populating vertex and index data:

#version 460
#extension GL_EXT_mesh_shader : require

layout(local_size_x = 32) in;
layout(triangles, max_vertices = 3, max_primitives = 1) out;

out gl_MeshPerVertexEXT {
    vec4 gl_Position;
} gl_MeshVerticesEXT[];

void main() {
    // Determine dynamic visibility (e.g., dynamic LOD or culling check)
    bool isVisible = true; 

    uint numVertices = isVisible ? 3 : 0;
    uint numPrimitives = isVisible ? 1 : 0;

    // Declare dynamic outputs prior to writing output arrays
    SetMeshOutputsEXT(numVertices, numPrimitives);

    if (!isVisible) {
        return;
    }

    // Populate data safely within bounds
    if (gl_LocalInvocationIndex == 0) {
        gl_PrimitiveTriangleIndicesEXT[0] = uvec3(0, 1, 2);
    }

    if (gl_LocalInvocationIndex < 3) {
        gl_MeshVerticesEXT[gl_LocalInvocationIndex].gl_Position = vec4(0.0);
    }
}

Through this mechanism, SetMeshOutputsEXT grants mesh shaders fine-grained control over workload amplification and dynamic data output directly on the GPU.