How Mesh Shaders Stop Vertex Cache Thrashing

Traditional graphics pipelines rely on fixed-function input assembly and small, hardware-managed post-transform caches to reuse shaded vertices, which frequently thrash when processing complex or poorly sorted geometry. Mesh shaders eliminate this bottleneck by restructuring geometry processing into self-contained, cooperative threadgroups called meshlets. By shifting vertex reuse from an unpredictable hardware cache to predictable, on-chip shared memory, mesh shaders prevent redundant vertex shading and bypass the cache thrashing inherent to standard index-buffer rendering.

The Traditional Bottleneck: Fixed-Function Indexing and Cache Thrashing

In the classic GLSL vertex pipeline, geometry is drawn using indexed primitives via calls like glDrawElements. The GPU's hardware Input Assembler reads an index buffer to fetch vertex attributes and assemble triangles.

To prevent shading the same vertex multiple times when shared across adjacent triangles, GPUs employ an on-chip Post-Transform Cache (typically a small FIFO or LRU cache holding between 16 to 64 vertices). If a vertex index is encountered that still resides in this cache, the hardware reuses the cached output rather than running the vertex shader again.

However, this mechanism creates severe performance pitfalls:

  • Capacity Limits: If mesh topology is dense or irregular, the distance between index references (the reuse distance) often exceeds the cache size, evicting vertices before their next use.
  • Order Dependence: The efficiency of the cache relies heavily on offline index-buffer optimization tools (such as Forsyth or Tipsify algorithms). Unoptimized or dynamically generated geometry causes cache misses.
  • Cache Thrashing: When indices continually evict entries that are needed immediately afterward, the pipeline experiences cache thrashing. This forces the GPU to re-evaluate vertex shaders multiple times for the identical vertex, multiplying shading cost and saturating memory bandwidth.

The Mesh Shader Paradigm: Cluster-Based Geometry

Mesh shaders replace the fixed-function Input Assembler, Vertex Shader, and Geometry Shader with a flexible, compute-like execution model. Geometry is partitioned ahead of time into small sub-graphs called meshlets (or clusters), typically bounded to a maximum of 64 vertices and 126 primitives.

Instead of processing a continuous stream of indices through a rigid pipeline, a mesh shader workgroup processes an entire meshlet cooperatively.

How Mesh Shaders Bypass Cache Thrashing

  1. Cooperative, Thread-Per-Vertex Execution
    Within a mesh shader workgroup, invocations correspond directly to the vertices and primitives defined in the meshlet. If a meshlet contains 64 unique vertices, exactly 64 threads are launched to process them. Each vertex is shaded exactly once, regardless of how many triangles within the meshlet reference it. This guarantees zero redundant vertex shading without relying on any hardware eviction heuristic.

  2. Explicit Shared Memory Over Implicit Caches
    Rather than relying on a black-box, fixed-size post-transform cache, the mesh shader writes transformed vertex data directly to on-chip Local Data Share (LDS) or High-Speed Shared Memory. The connectivity of the meshlet is defined via micro-indices that refer strictly to this local, on-chip array. Because all shared vertex data is stored in fast, addressable local memory for the duration of the workgroup, it cannot be prematurely evicted or thrashed by distant geometry.

  3. Elimination of Global Index Buffer Reads
    In the traditional pipeline, index lookups are scattered across a global buffer, causing non-coalesced memory reads that further degrade cache efficiency. Mesh shaders fetch compact, localized data streams. The relationship between vertices and primitives is localized entirely within the meshlet payload, decoupling primitive assembly from global memory state.

  4. Guaranteed Local Topology
    Because meshlets are bounded by definition, the vertex reuse distance is physically constrained to the maximum size of the meshlet. Topological complexity cannot blow out a hardware cache because the geometry never exceeds the localized footprint assigned to the computing workgroup.

By moving from a streaming, index-driven model to a localized, compute-driven cluster model, mesh shaders replace probabilistic hardware caching with deterministic algorithmic reuse, entirely bypassing the mechanics that cause vertex index cache thrashing.