DirectX 11 Tessellation Pipeline Explained

DirectX 11 introduced hardware-accelerated tessellation to dynamically generate detailed surface geometry from coarse base meshes entirely on the GPU. This mechanism relies on three distinct pipeline stages inserted between the traditional Vertex Shader and Geometry Shader: the Hull Shader, the Tessellator, and the Domain Shader. Together, these stages enable efficient Level of Detail (LOD) management, reduce memory bandwidth by avoiding the storage of dense polygonal meshes, and allow realistic displacement mapping to be computed in real time.

The Hull Shader Stage

The Hull Shader is a programmable stage that operates directly on input control points representing a surface patch, such as a Bézier surface or Catmull-Clark subdivision patch. It serves two main functions, split across two phases:

  1. Control Point Phase: The shader processes the incoming control points from the Vertex Shader, optionally transforming or generating new control points for the patch basis.
  2. Patch Constant Phase: The shader calculates per-patch data, most critically the Tessellation Factors (TessFactors). These factors tell the hardware how finely to subdivide the patch along its edges and its interior.

Developers use the Hull Shader to implement dynamic LOD algorithms. For instance, patches closer to the camera or with sharp curvature can be assigned higher tessellation factors, while distant or flat surfaces are assigned lower factors to conserve processing power.

The Tessellator Stage

The Tessellator is a fixed-function hardware stage positioned immediately after the Hull Shader. It performs the raw computational subdivision.

The Tessellator does not process 3D spatial coordinates or mesh data directly. Instead, it operates in a normalized parameter space (such as a 2D quad, triangle, or isoline) and uses the TessFactors provided by the Hull Shader. Based on these factors, the Tessellator generates a topological mesh of new points and triangles within the normalized domain, outputting a collection of \((u, v)\) or \((u, v, w)\) barycentric coordinates.

The Domain Shader Stage

The Domain Shader is the final, programmable stage of the tessellation triad. It is invoked once for every single vertex generated by the fixed-function Tessellator.

The Domain Shader receives the parametric coordinates from the Tessellator alongside the original patch control points and patch constant data from the Hull Shader. Its primary responsibility is to calculate the final 3D world-space position, normal, and texture coordinates of each new vertex.

At this stage, techniques like displacement mapping are applied. The Domain Shader samples a height map using the interpolated texture coordinates and offsets the calculated vertex position along the surface normal. The resulting high-resolution geometry is then forwarded down the rendering pipeline to the Geometry Shader (if active) or directly to the Rasterizer.