Using gl_SampleMaskIn for Coverage Masks in GLSL

This article explores the built-in GLSL variable gl_SampleMaskIn, explaining its role in multisample anti-aliasing (MSAA) pipelines. Readers will learn how this variable exposes the primitive coverage generated during rasterization, how its bitmask structure represents individual sample points, and how developers can utilize this pre-shading data to implement advanced rendering techniques such as custom resolves and order-independent transparency.

What is gl_SampleMaskIn?

In GLSL, gl_SampleMaskIn is a built-in fragment shader input variable declared as an array of integers:

in int gl_SampleMaskIn[];

Its primary role is to inform the fragment shader which sample points within a pixel are covered by the primitive being rendered, before any fragment shading calculations occur. When multisampling is enabled, the graphics pipeline performs rasterization at the sub-pixel sample level. The result of this geometric coverage test is captured in gl_SampleMaskIn.

Structure and Bit Representation

gl_SampleMaskIn functions as a bitfield where each bit corresponds to a specific sample index:

  • Bit Indexing: Bit \(n\) of gl_SampleMaskIn[w] represents the coverage state of sample \(32 \times w + n\).
  • Value State: If bit \(n\) is 1, sample \(n\) is covered by the incoming primitive. If it is 0, the sample is not covered.
  • Array Size: The size of the array is determined by the maximum number of samples supported divided by 32, rounded up. For typical multisampling modes (2x, 4x, 8x, or 16x MSAA), only the first element, gl_SampleMaskIn[0], is populated.

For example, in a 4x MSAA setup, a fully covered pixel results in the lowest four bits of gl_SampleMaskIn[0] being set (0x0000000F). If a triangle edge crosses the pixel and covers only sample indices 0 and 2, the mask reads 0x00000005 (binary ...0101).

Pipeline Timing: When is the Mask Determined?

gl_SampleMaskIn reflects coverage determined during the rasterization stage:

  1. Geometric Coverage: The hardware evaluates which MSAA sample positions fall inside the primitive boundary.
  2. Early Per-Fragment Tests: If early depth, stencil, or scissor tests are active, samples that fail these tests may also be removed from the mask before shading.
  3. Shader Invocation: The fragment shader executes. Regardless of whether the shader runs per-pixel or per-sample, gl_SampleMaskIn provides the comprehensive coverage state of the entire pixel.

Importantly, gl_SampleMaskIn is read-only. It captures the initial hardware-determined visibility before any shader logic executes or any modification via the output mask gl_SampleMask[] occurs.

Key Applications

Accessing hardware coverage masks prior to shading allows developers to bypass standard hardware resolve paths and implement custom sub-pixel algorithms:

1. Custom MSAA Resolves and Deferred Shading

Standard deferred shading does not integrate cleanly with MSAA because running lighting per-sample is computationally expensive. By querying gl_SampleMaskIn, a deferred shader can detect complex geometry edges (where the mask is neither fully empty nor fully set) and apply expensive per-sample shading only to edge pixels, while shading interior pixels once per fragment.

2. Alpha-to-Coverage Analysis

When using alpha-to-coverage techniques, the alpha channel is converted into a dithered coverage mask. Inspecting gl_SampleMaskIn lets shaders evaluate how hardware coverage interacts with alpha-tested geometry before generating color outputs.

3. Selective Discarding via gl_SampleMask

While gl_SampleMaskIn reports incoming coverage, the fragment shader can modify the final coverage mask by writing to gl_SampleMask[]. By reading gl_SampleMaskIn, applying a bitwise operation (such as AND), and writing the result to gl_SampleMask, a shader can disable specific sub-pixel samples without discarding the entire fragment.

void main() {
    // Read the incoming hardware coverage
    int coverage = gl_SampleMaskIn[0];

    // Mask out sample 0 dynamically
    gl_SampleMask[0] = coverage & ~1;
}

Summary

gl_SampleMaskIn bridges the gap between fixed-function rasterization and programmable fragment processing. By providing a bit-accurate view of sub-pixel primitive coverage before shading begins, it gives developers the fine-grained visibility data necessary to optimize anti-aliasing passes, implement complex resolve strategies, and manage sample coverage programmatically.