How DirectX Handles MSAA During Rasterization

This article explains how the DirectX graphics pipeline processes multi-sample anti-aliasing (MSAA) during the rasterization stage. In DirectX, MSAA reduces geometric edge aliasing by decoupling geometric coverage testing from pixel shading frequency. The rasterizer evaluates primitive coverage across multiple distinct sub-pixel sample locations, executes the pixel shader only once per covered pixel under standard configurations, and broadcasts the resulting color to all covered samples that pass depth and stencil tests.

Sub-Pixel Coverage Testing

When a primitive arrives at the rasterizer state with MSAA enabled (such as 2x, 4x, or 8x MSAA), the rasterizer evaluates geometric coverage at multiple fixed sub-pixel positions instead of a single point at the pixel center. The rasterizer tests the triangle's edges against each sample coordinate within the pixel's bounds.

A bitmask, known as the coverage mask, is generated for each pixel. Each bit in the mask corresponds to a sample point, indicating whether that specific sub-pixel falls inside or outside the primitive's boundaries.

Per-Sample Depth and Stencil Testing

Depth and stencil testing in an MSAA-enabled DirectX pipeline occurs at sub-pixel sample frequency, not pixel frequency. Each sample point maintains its own independent depth and stencil values in the multi-sampled depth-stencil buffer.

The rasterizer interpolates the primitive's depth (\(z\)) and \(w\) values individually for each sample point that passed the coverage test. Early-Z or late-Z testing evaluates each sample against the existing values in the depth-stencil target. If a sample fails the depth test, its corresponding bit in the coverage mask is discarded.

Decoupling Shading from Coverage

The defining efficiency of MSAA in DirectX compared to Super-Sample Anti-Aliasing (SSAA) is the separation of coverage from shading. Unless Sample-Frequency Shading (SV_SampleIndex) is explicitly triggered in the pixel shader, the rasterizer invokes the pixel shader only once per primitive for any pixel where at least one sample is covered.

By default, the pixel shader evaluates inputs at the pixel center. However, if the pixel center is not covered by the primitive, this can lead to sampling textures or attributes outside the triangle's boundaries, causing visual artifacts. To resolve this, DirectX supports centroid sampling. When attributes are declared with the centroid modifier in HLSL, the rasterizer evaluates the attribute values at a position within the area bounded by the primitive and the pixel, typically the average center of all covered samples.

Output and Sample Mask Application

Once the pixel shader finishes execution, it outputs a single color value. The rasterizer and output-merger apply this color to every sample location whose bit remains active in the coverage mask and has passed the depth/stencil test.

DirectX also provides pipeline states such as SampleMask in the blend state description, which allows developers to globally mask out specific sub-pixel samples across all render operations.

Final Resolve

The result of the rasterization process is a multi-sampled render target where each pixel stores \(N\) distinct color, depth, and stencil values (where \(N\) is the sample count). These samples remain discrete until the application resolves the multi-sampled resource into a non-multisampled resource using the DirectX API call ResolveSubresource or custom compute/pixel shader resolve passes, which average the sub-pixel colors to produce smooth, anti-aliased edges.