Understanding gl_SampleMask in GLSL Fragment Shaders

In multisample rendering, the gl_SampleMask built-in output array in GLSL allows a fragment shader to programmatically control which sub-pixel samples are written to the framebuffer. This article explains the function of gl_SampleMask, how it interacts with the GPU's rasterization pipeline, and common scenarios where developers use it to customize multisample anti-aliasing (MSAA).

Definition and Functionality

gl_SampleMask is a built-in fragment shader output variable defined as an array of 32-bit integers (out int gl_SampleMask[]). Its primary role is to set a bitmask that dictates sample coverage for the currently processed fragment.

Each bit in the mask corresponds to an individual sub-pixel sample:

  • Bit \(n\) in gl_SampleMask[0] determines coverage for sample \(n\) (from sample 0 up to sample 31).
  • If an implementation supports more than 32 samples per pixel, subsequent array elements control higher-indexed samples (e.g., gl_SampleMask[1] covers samples 32 through 63).

When a bit is set to 1, the corresponding sample remains eligible to be written to the framebuffer. When a bit is set to 0, that sample is discarded, preventing color, depth, and stencil updates for that specific sample location.

Interaction with Hardware Rasterization

Writing to gl_SampleMask does not grant the shader the ability to revive samples outside the primitive's geometric boundaries. Instead, the final coverage mask is determined by performing a bitwise AND operation between:

  1. The hardware-generated coverage mask produced during polygon rasterization.
  2. The values written by the fragment shader to gl_SampleMask.
  3. Other coverage modifiers, such as alpha-to-coverage, if enabled.

Consequently, gl_SampleMask acts as a subtractive filter. It can turn covered samples off, but it cannot activate samples that were not originally covered by the geometric primitive being rasterized.

Usage in GLSL

To use gl_SampleMask, multisample anti-aliasing must be enabled in the OpenGL context. Within the fragment shader, values are assigned directly to the array elements:

#version 450 core

out vec4 fragColor;

void main()
{
    fragColor = vec4(1.0, 0.0, 0.0, 1.0);

    // Disable sample 0 and sample 2, keep sample 1 and sample 3 active
    // Binary: 0000...1010 -> 0x0A
    gl_SampleMask[0] = 0x0A; 
}

If the shader does not write to gl_SampleMask, the default behavior retains all samples determined by the fixed-function rasterizer.

Common Use Cases

  • Custom Screen-Door and Stippled Transparency: Developers can simulate partial transparency by dynamically assigning alternating sample patterns across neighboring pixels, avoiding the need for strict back-to-front primitive sorting.
  • Custom Alpha-to-Coverage: While OpenGL provides automatic alpha-to-coverage, manual assignment via gl_SampleMask gives precise control over which specific dither patterns are applied at varying opacity levels.
  • Pixel-Art Scaling and Custom Filtering: When rendering low-resolution textures into multisampled framebuffers, developers can selectively mask out specific samples to retain crisp geometric edges or prevent unwanted blending artifacts.