HBAO Tangent Angle Horizon Search in GLSL

This article explains how to evaluate the tangent angle horizon search of Horizon-Based Ambient Occlusion (HBAO) inside an OpenGL Shading Language (GLSL) fragment shader. It covers reconstructing view-space positions, projecting surface normals into search directions to determine the tangent angle, marching along screen-space rays to identify the maximum horizon elevation, and applying distance attenuation to compute physically plausible ambient occlusion.

Mathematical Principle of HBAO

Standard Screen-Space Ambient Occlusion (SSAO) samples points uniformly in a sphere or hemisphere, introducing noise and self-occlusion artifacts. HBAO models ambient occlusion as an integral over the hemisphere by casting a set of directional rays across the depth buffer.

For each direction, HBAO determines:

  1. The Tangent Angle (\(\theta\) or \(t\)): The elevation angle of the surface's tangent plane along the search direction.
  2. The Horizon Angle (\(h\)): The maximum elevation angle formed by neighboring geometry relative to the current surface point.

The occlusion contribution for a single ray direction is defined by the difference between the sine of the maximum horizon angle and the sine of the tangent angle:

\[A = \max(0, \sin(h) - \sin(t))\]

View-Space Position Reconstruction

Before evaluating angles, depth values must be converted to view-space coordinates:

vec3 GetViewPos(vec2 uv, sampler2D depthTexture, mat4 invProjMatrix) {
    float depth = texture(depthTexture, uv).r;
    vec4 clipSpace = vec4(uv * 2.0 - 1.0, depth * 2.0 - 1.0, 1.0);
    vec4 viewSpace = invProjMatrix * clipSpace;
    return viewSpace.xyz / viewSpace.w;
}

Tangent Angle Evaluation

For a given screen-space search direction \(\vec{\omega} = (\cos\phi, \sin\phi)\), the surface tangent angle defines the baseline above which geometry causes occlusion.

Given a view-space normal \(\mathbf{N} = (N_x, N_y, N_z)\) and a screen-space ray direction \(\mathbf{D} = (D_x, D_y)\):

  1. The derivative of the surface depth in direction \(\mathbf{D}\) is determined by the tangent vector \(\mathbf{T}\) along the surface plane.
  2. Because \(\mathbf{N} \cdot \mathbf{T} = 0\), the elevation angle of the tangent plane can be expressed as:

\[\tan(t) = -\frac{N_x D_x + N_y D_y}{N_z}\]

Using the identity \(\sin(t) = \frac{\tan(t)}{\sqrt{1 + \tan^2(t)}}\), the tangent elevation sine is obtained directly without calling costly inverse trigonometric functions.

float ComputeTangentSin(vec3 normal, vec2 dir) {
    // Tangent slope along the search vector dir
    float tanT = -(normal.x * dir.x + normal.y * dir.y) / normal.z;
    return tanT / sqrt(1.0 + tanT * tanT);
}

Horizon Ray-Marching Loop

Along each direction ray, step outward in screen space by regular increments. A random noise texture or interleaved gradient noise is typically added to jitter the initial step offset to reduce banding.

For every step:

  1. Sample depth and reconstruct sample position \(\mathbf{S}\).
  2. Construct the horizon vector \(\mathbf{H} = \mathbf{S} - \mathbf{P}\), where \(\mathbf{P}\) is the current fragment position.
  3. Compute the elevation sine of this vector:

\[\sin(h) = \frac{H_z}{\|\mathbf{H}\|}\]

  1. Apply an attenuation function based on Euclidean distance \(\|\mathbf{H}\|\) to discard surfaces outside a defined radius \(R\).
  2. Update the running maximum \(\sin(h)\).

Complete GLSL Horizon Search Implementation

The following fragment shader function demonstrates the core horizon march for a single direction:

// Parameters
const int   NUM_STEPS    = 6;
const float RADIUS       = 1.5;     // View-space search radius
const float ATTEN_FACTOR = 1.0;     // Falloff rate

float Falloff(float dist, float radius) {
    // Quadratic distance attenuation
    return clamp(1.0 - (dist * dist) / (radius * radius), 0.0, 1.0);
}

float EvaluateHBAODirection(
    vec2 texCoord, 
    vec2 screenDir, 
    vec3 viewPos, 
    vec3 viewNormal, 
    float stepSizeUV, 
    float jitter, 
    sampler2D depthTex, 
    mat4 invProj) 
{
    // 1. Calculate the tangent sine threshold
    float sinTangent = ComputeTangentSin(viewNormal, screenDir);
    float maxSinHorizon = sinTangent;

    // 2. March along the screen-space direction
    for (int i = 1; i <= NUM_STEPS; ++i) {
        // Apply jitter to sampling distance
        float stepDist = (float(i) + jitter) * stepSizeUV;
        vec2 sampleUV = texCoord + screenDir * stepDist;

        // Skip samples outside the viewport
        if (sampleUV.x < 0.0 || sampleUV.x > 1.0 || sampleUV.y < 0.0 || sampleUV.y > 1.0)
            break;

        vec3 samplePos = GetViewPos(sampleUV, depthTex, invProj);
        vec3 horizonVec = samplePos - viewPos;
        float dist = length(horizonVec);

        // 3. Compute horizon elevation sine
        // Assumes view space looks along -Z (Z is negative in front of camera)
        float sinHorizon = -horizonVec.z / dist;

        // 4. Distance falloff check
        float atten = Falloff(dist, RADIUS);

        // 5. Update maximum horizon angle
        if (sinHorizon > maxSinHorizon && dist < RADIUS) {
            maxSinHorizon = mix(maxSinHorizon, sinHorizon, atten);
        }
    }

    // 6. Compute occlusion contribution from this slice
    return clamp(maxSinHorizon - sinTangent, 0.0, 1.0);
}

Integration and Performance Considerations

  • Coordinate Orientation: View-space conventions vary. When the camera points along \(-Z\), depths are negative, meaning elevation slopes require matching sign inversions.
  • Early Exits: Discard rays whose screen-space step length falls below sub-pixel thresholds when surfaces are viewed at steep grazing angles.
  • Trigonometric Elimination: Computing \(\sin(h)\) directly via normalization (\(H_z / \|\mathbf{H}\|\)) avoids all expensive calls to atan or asin.