Passing Bounding Volumes with taskPayloadSharedEXT

The taskPayloadSharedEXT storage qualifier acts as the dedicated communication interface between task shaders and mesh shaders in the Vulkan and OpenGL mesh shading pipelines. By allowing user-defined structures to reside in high-speed, on-chip shared memory, it enables the task shader to perform coarse culling operations and seamlessly forward filtered bounding volume metadata, instance data, and meshlet offsets directly to downstream mesh shaders without round-tripping through device memory.

The Role of taskPayloadSharedEXT

In the GL_EXT_mesh_shader extension, task shaders operate as a pre-amplification stage capable of dynamically spawning mesh shader workgroups. Communication across this stage boundary cannot rely on standard interface blocks or typical invocation outputs. Instead, the taskPayloadSharedEXT qualifier defines a shared memory allocation that bridges the execution boundary between a single task shader workgroup and the mesh shader workgroups it spawns via EmitMeshTasksEXT().

Structuring Bounding Volume Metadata

Bounding volume metadata—such as bounding sphere centers and radii, or axis-aligned bounding box (AABB) minimum and maximum extents—is typically organized inside a custom struct. Both the task shader and the mesh shader must declare an identical struct and instantiate it using the taskPayloadSharedEXT qualifier.

// Shared definition in both Task and Mesh shaders
struct BoundingBox {
    vec3 minExtents;
    vec3 maxExtents;
};

struct TaskPayload {
    BoundingBox subBounds[32];
    uint meshletIDs[32];
};

taskPayloadSharedEXT TaskPayload payload;

When passing bounding metadata, this struct stores either refined bounding volumes for secondary culling inside the mesh shader or local-space to world-space transformation matrices used to evaluate pre-culled bounding hierarchies.

Data Flow from Task to Mesh Shader

The interaction follows a strict producer-consumer model:

  1. Task Shader Execution (Producer): A task shader workgroup processes a cluster of meshlets or objects. A single invocation, or the workgroup cooperatively, performs early frustum or occlusion culling using coarse bounding volumes. Surviving meshlets and their corresponding refined bounding data are written directly into payload.

  2. Synchronization: Because the payload is shared across all invocations within the task workgroup, memory writes must be synchronized using memoryBarrierShared() and barrier() before emitting downstream tasks. This prevents race conditions where some invocations write to the payload while others trigger execution.

  3. Spawning Mesh Workgroups: The task shader invokes EmitMeshTasksEXT(groupCountX, groupCountY, groupCountZ). At this point, the current state of the payload is committed for the spawned mesh shader workgroups.

  4. Mesh Shader Consumption (Consumer): Every mesh shader workgroup generated by that specific task workgroup receives the payload as read-only data. Mesh shader invocations can read the bounding volumes from payload.subBounds[gl_WorkGroupID.x] to perform fine-grained frustum culling, cone-culling (backface testing), or LOD-level adjustments before generating vertices and primitives.

Memory Constraints and Considerations

While taskPayloadSharedEXT provides high bandwidth, it is constrained by hardware-defined size limits (queried via maxTaskPayloadSize). Transmitting bounding metadata requires attention to memory alignment and footprint:

  • Payload Footprint: Storing large arrays of full AABBs (six 32-bit floats each) can quickly exceed size budgets and limit execution occupancy. Quantizing bounding extents (e.g., using 16-bit floats or center-radius representations) helps preserve on-chip cache space.
  • Read-Only Invariance: Mesh shaders cannot modify taskPayloadSharedEXT variables; attempts to write to the payload from a mesh shader will result in a compile-time error.
  • Workgroup Mapping: The payload is statically allocated per task workgroup. Each dispatched mesh workgroup references the same payload instance, meaning dynamic indexing should rely on built-in variables like gl_WorkGroupID.x to map each mesh workgroup to its corresponding slice of bounding metadata.