Hierarchical Z-Buffer Occlusion Culling in GLSL

This article explains the algorithm for implementing hierarchical Z-buffer (Hi-Z) occlusion culling of bounding boxes within a GLSL compute shader. Hi-Z culling accelerates rendering pipelines by testing the projected screen-space bounding box of an object against a pre-generated depth mipmap pyramid, allowing the GPU to reject fully occluded instances before submitting them to the rasterizer.

Overview of the Hi-Z Pipeline

Hi-Z occlusion culling runs in a compute pass prior to the main geometry rendering. It consists of two primary phases:

Hi-Z Depth Pyramid Construction: The depth buffer from the previous frame (or an early-Z prepass) is downsampled into a mipmap chain. Each downsampled texel stores the conservative depth value of its corresponding \(2 \times 2\) texels in the higher-resolution level (the maximum depth for standard \([0, 1]\) depth, or minimum depth for reverse-Z).
Compute Cull Pass: A compute shader processes an array of candidate bounding boxes, transforms them into screen space, determines the appropriate mip level based on their screen-space size, samples the Hi-Z pyramid, and writes visible instance indices into an indirect draw buffer.

The Culling Algorithm Steps

1. Transform the Bounding Box to Clip Space

Given an Axis-Aligned Bounding Box (AABB) defined by its minimum and maximum object-space coordinates, transform all eight vertices by the Model-View-Projection (MVP) matrix into homogeneous clip space.

P_clip = MVP * P_obj

If any point has \(W \le 0\), clip against the near plane or mark the box as visible to prevent projection artifacts.

2. Project to Normalized Device Coordinates (NDC)

Perform the perspective divide for all 8 vertices:

\[\mathbf{P}_{ndc} = \frac{\mathbf{P}_{clip}.xyz}{\mathbf{P}_{clip}.w}\]

Convert the NDC coordinates (\([-1, 1]\)) into screen-space UV coordinates (\([0, 1]\)):

\[\mathbf{P}_{uv} = \mathbf{P}_{ndc}.xy \times 0.5 + 0.5\]

3. Compute the Screen-Space 2D AABB

Find the axis-aligned min and max in UV space, clamped to the screen boundaries:

\[\mathbf{UV}_{min} = \text{clamp}(\min(\mathbf{P}_{uv}), 0.0, 1.0)\] \[\mathbf{UV}_{max} = \text{clamp}(\max(\mathbf{P}_{uv}), 0.0, 1.0)\]

Find the minimum depth (closest point to the camera) of the bounding box:

\[Z_{min} = \min(\mathbf{P}_{ndc}.z)\]

(Note: In a standard depth buffer where 0 is near and 1 is far, \(Z_{min}\) represents the closest geometry. In reverse-Z, you query the maximum value.)

4. Select the Hi-Z Mip Level

Calculate the size of the 2D bounding box in pixels:

\[\text{Size}_{px} = (\mathbf{UV}_{max} - \mathbf{UV}_{min}) \times \text{ViewportDimensions}\]

Select a mip level where the 2D bounding box covers a footprint of approximately \(2 \times 2\) texels:

\[\text{mip} = \text{ceil}\left(\log_2\left(\max(\text{Size}_{px}.x, \text{Size}_{px}.y)\right)\right)\]

Sampling at this mipmap level guarantees that a small number of samples (typically \(2 \times 2\) texels) covers the entire screen footprint of the bounding box.

5. Fetch Depth from the Hi-Z Buffer

Sample the selected mip level across the footprint. Using a \(2 \times 2\) grid covering the corners of the UV bounding box ensures no occluders inside the region are missed:

float d0 = textureLod(u_HiZBuffer, vec2(uvMin.x, uvMin.y), mip).r;
float d1 = textureLod(u_HiZBuffer, vec2(uvMax.x, uvMin.y), mip).r;
float d2 = textureLod(u_HiZBuffer, vec2(uvMin.x, uvMax.y), mip).r;
float d3 = textureLod(u_HiZBuffer, vec2(uvMax.x, uvMax.y), mip).r;

Find the conservative depth value across the four samples:

Standard Depth (\([0, 1]\), 0 is near): \(Z_{hiz} = \max(d0, \max(d1, \max(d2, d3)))\)
Reverse-Z (\([1, 0]\), 1 is near): \(Z_{hiz} = \min(d0, \min(d1, \min(d2, d3)))\)

6. Perform the Depth Test

Compare the bounding box depth to the Hi-Z depth:

Standard Depth: If \(Z_{min} > Z_{hiz}\), the bounding box is strictly behind the geometry represented in the Hi-Z buffer and is occluded.
Reverse-Z: If \(Z_{max} < Z_{hiz}\), the bounding box is occluded.

If visible, append the instance index into the draw buffer using atomic counters.

GLSL Compute Implementation

The following compute shader demonstrates the core culling routine for standard \([0, 1]\) depth.

#version 460 core

layout(local_size_x = 64) in;

struct InstanceData {
    mat4 modelMatrix;
    vec3 aabbMin;
    vec3 aabbMax;
};

layout(std430, binding = 0) readonly buffer InstanceBuffer {
    InstanceData instances[];
};

layout(std430, binding = 1) writeonly buffer VisibleBuffer {
    uint visibleIndices[];
};

layout(binding = 2) uniform atomic_uint u_VisibleCount;

layout(binding = 0) uniform sampler2D u_HiZBuffer;

uniform mat4 u_ViewProjection;
uniform vec2 u_ViewportSize;
uniform uint u_TotalInstances;

void main() {
    uint id = gl_GlobalInvocationID.x;
    if (id >= u_TotalInstances) return;

    InstanceData inst = instances[id];
    mat4 mvp = u_ViewProjection * inst.modelMatrix;

    vec3 boxCorners[8] = vec3[8](
        vec3(inst.aabbMin.x, inst.aabbMin.y, inst.aabbMin.z),
        vec3(inst.aabbMax.x, inst.aabbMin.y, inst.aabbMin.z),
        vec3(inst.aabbMin.x, inst.aabbMax.y, inst.aabbMin.z),
        vec3(inst.aabbMax.x, inst.aabbMax.y, inst.aabbMin.z),
        vec3(inst.aabbMin.x, inst.aabbMin.y, inst.aabbMax.z),
        vec3(inst.aabbMax.x, inst.aabbMin.y, inst.aabbMax.z),
        vec3(inst.aabbMin.x, inst.aabbMax.y, inst.aabbMax.z),
        vec3(inst.aabbMax.x, inst.aabbMax.y, inst.aabbMax.z)
    );

    vec3 minNDC = vec3(1.0);
    vec3 maxNDC = vec3(-1.0);
    bool behindNearPlane = false;

    for (int i = 0; i < 8; ++i) {
        vec4 clipPos = mvp * vec4(boxCorners[i], 1.0);
        
        if (clipPos.w <= 0.0) {
            behindNearPlane = true;
        }

        vec3 ndc = clipPos.xyz / clipPos.w;
        minNDC = min(minNDC, ndc);
        maxNDC = max(maxNDC, ndc);
    }

    // Conservative visibility: if any vertex intersects or is behind the near plane
    if (behindNearPlane) {
        uint slot = atomicCounterIncrement(u_VisibleCount);
        visibleIndices[slot] = id;
        return;
    }

    // Frustum cull
    if (any(lessThan(maxNDC, vec3(-1.0))) || any(greaterThan(minNDC, vec3(1.0)))) {
        return;
    }

    // Convert NDC to UV [0, 1]
    vec2 minUV = clamp(minNDC.xy * 0.5 + 0.5, 0.0, 1.0);
    vec2 maxUV = clamp(maxNDC.xy * 0.5 + 0.5, 0.0, 1.0);
    float boxMinZ = clamp(minNDC.z * 0.5 + 0.5, 0.0, 1.0);

    // Compute mip level matching the screen-space bounding box size
    vec2 sizeUV = (maxUV - minUV) * u_ViewportSize;
    float maxDim = max(sizeUV.x, sizeUV.y);
    float mip = ceil(log2(max(maxDim, 1.0)));

    // Sample the 4 corners of the bounding box footprint at the selected mip level
    float d0 = textureLod(u_HiZBuffer, vec2(minUV.x, minUV.y), mip).r;
    float d1 = textureLod(u_HiZBuffer, vec2(maxUV.x, minUV.y), mip).r;
    float d2 = textureLod(u_HiZBuffer, vec2(minUV.x, maxUV.y), mip).r;
    float d3 = textureLod(u_HiZBuffer, vec2(maxUV.x, maxUV.y), mip).r;

    // Conservative max depth in Hi-Z for standard depth buffer
    float maxHiZ = max(max(d0, d1), max(d2, d3));

    // Occlusion check
    if (boxMinZ <= maxHiZ) {
        uint slot = atomicCounterIncrement(u_VisibleCount);
        visibleIndices[slot] = id;
    }
}