GL_EXT_post_depth_coverage and gl_SampleMaskIn in GLSL

The GL_EXT_post_depth_coverage extension fundamentally alters the gl_SampleMaskIn input in GLSL by updating its bitmask to reflect sample coverage after the depth and stencil tests have executed, rather than before. Under standard pipeline rules, gl_SampleMaskIn reflects the raw geometric coverage generated by rasterization. By modifying this behavior, the extension allows fragment shaders running in multisample anti-aliasing (MSAA) modes to identify exactly which sub-pixel samples survived depth and stencil testing, enabling efficient custom resolves, advanced visibility algorithms, and reduced shading overhead.

Default Behavior of gl_SampleMaskIn

In core OpenGL and standard GLSL, gl_SampleMaskIn is an array of integers serving as a bitmask where each bit corresponds to a sample within the fragment.

Under normal execution:

• The rasterizer determines which sub-pixel samples are covered by the incoming primitive geometry.
• This coverage mask is passed directly to the fragment shader via gl_SampleMaskIn.
• Even if early depth testing is active, the specification dictates that gl_SampleMaskIn indicates coverage before the depth and stencil tests are evaluated. Occluded samples that are about to fail the depth test still have their corresponding bits set to 1.

How GL_EXT_post_depth_coverage Changes Visibility

When GL_EXT_post_depth_coverage is enabled, the timing of the coverage mask calculation is shifted. Instead of receiving the pre-depth rasterization mask:

1. Depth and Stencil Evaluation First: The GPU evaluates the depth and stencil tests for all covered samples prior to running the shader body.
2. Mask Modification: Any sample that fails the depth or stencil test has its corresponding bit cleared to 0.
3. Updated Input: gl_SampleMaskIn is populated only with the samples that are both covered by the primitive and successfully pass the depth and stencil tests.

If all samples for a given fragment fail the depth and stencil tests, the fragment is discarded entirely, and the fragment shader does not execute.

Shader Declaration and Usage

To use this functionality in GLSL, the extension must be enabled, and the input layout qualifier must be applied:

#version 450 core
#extension GL_EXT_post_depth_coverage : require

// Enable post-depth coverage for the fragment shader
layout(post_depth_coverage) in;

// Post-depth coverage inherently relies on early tests
layout(early_fragment_tests) in;

out vec4 fragColor;

void main() {
    // gl_SampleMaskIn[0] now contains only surviving samples
    int survivingSamples = gl_SampleMaskIn[0];

    // If a sample failed the depth test, its bit is 0
    if ((survivingSamples & 1) != 0) {
        // Sample 0 survived depth/stencil testing
    }

    fragColor = vec4(1.0);
}

Key Technical Benefits

• Custom MSAA Resolves: Custom multi-sample resolve passes can avoid processing occluded sample data, improving memory access efficiency.
• Order-Independent Transparency (OIT): Per-pixel linked lists and visibility buffers can accurately link only the visible sub-samples, avoiding the storage of completely occluded fragments.
• Accurate Sub-Pixel Shading: Shaders performing per-sample computational workloads can loop over gl_SampleMaskIn bits and selectively skip samples that failed the depth test, avoiding redundant calculations.