Export Dynamic Stencil Values with gl_FragStencilRefARB

The gl_FragStencilRefARB built-in variable allows fragment shaders to dynamically generate and export per-fragment stencil reference values directly to the OpenGL stencil buffer. Traditionally, stencil reference values are immutable across an entire draw call, set globally by the host application via API functions like glStencilFunc. By enabling the GL_ARB_shader_stencil_export extension, developers gain programmatic control over the stencil reference value per fragment, bypassing static CPU-bound constraints to support advanced rendering techniques like dynamic masking, deferred shading tagging, and procedural stencil generation.

The Limitation of Static Reference Values

In standard OpenGL pipelines, the stencil test compares a static reference value against the current value in the stencil buffer using a specified comparison function:

glStencilFunc(GL_EQUAL, refValue, mask);

Because refValue is set on the CPU, every fragment rasterized during that draw call shares the exact same reference value. To update the stencil buffer with multiple distinct IDs or mask regions, an application historically had to issue separate draw calls for each unique stencil value, introducing substantial CPU overhead and draw call batching inefficiencies.

Enabling Shader Stencil Export

To use dynamic stencil reference export in GLSL, the application must confirm hardware support for the GL_ARB_shader_stencil_export extension. Inside the fragment shader, the extension must be explicitly declared before use:

#version 330 core
#extension GL_ARB_shader_stencil_export : require

When enabled, the compiler exposes the output variable gl_FragStencilRefARB.

Syntax and Declaration

gl_FragStencilRefARB is an implicitly declared, built-in output variable with an integer type:

out int gl_FragStencilRefARB;

In the fragment shader, an integer value can be assigned to gl_FragStencilRefARB during execution. This value serves as the replacement reference value for subsequent stencil operations applied to that specific fragment:

#version 330 core
#extension GL_ARB_shader_stencil_export : require

in vec2 vTexCoord;
out vec4 fragColor;

uniform sampler2D uMaskTexture;

void main()
{
    fragColor = vec4(1.0, 0.0, 0.0, 1.0);

    // Compute a dynamic stencil value per-fragment
    int dynamicID = int(texture(uMaskTexture, vTexCoord).r * 255.0);

    // Export to the stencil test
    gl_FragStencilRefARB = dynamicID;
}

Pipeline Integration and Stencil Operations

When gl_FragStencilRefARB is written:

1. Comparison Test: If the stencil test evaluates against the reference value (e.g., using GL_EQUAL, GL_LESS), the per-fragment exported value replaces the static reference value defined by glStencilFunc.
2. Buffer Write: If the stencil test passes and the stencil operation specifies an action using the reference value (e.g., glStencilOp(..., ..., GL_REPLACE)), the value written into the stencil buffer is the value assigned to gl_FragStencilRefARB.

If the shader executes but conditionally omits writing to gl_FragStencilRefARB, or if the fragment is discarded using discard, standard stencil rules or discard logic apply.

Performance Considerations

Exporting stencil values from the fragment shader can impact graphics pipeline optimizations. Standard OpenGL implementations often leverage "Early Stencil" tests, which execute stencil and depth operations prior to running the fragment shader to eliminate occluded pixels early. Writing dynamic stencil values inside the fragment shader introduces a data dependency, generally forcing the GPU to execute the fragment shader first and delay stencil operations to the late fragment test stage.