Calculate Vector Rejection in GLSL

This article explains how to compute the orthogonal vector rejection component of one 3D vector from another using OpenGL Shading Language (GLSL). Vector rejection yields the component of a vector that is strictly perpendicular (orthogonal) to a reference vector. Below, you will find the mathematical principle behind the operation along with optimized GLSL implementations for both arbitrary and normalized vectors.

Mathematical Foundation

Given an input vector \(\vec{a}\) and a reference vector \(\vec{b}\), \(\vec{a}\) can be decomposed into two orthogonal components:

\[\vec{a} = \vec{a}_{\parallel} + \vec{a}_{\perp}\]

• \(\vec{a}_{\parallel}\) is the vector projection of \(\vec{a}\) onto \(\vec{b}\).
• \(\vec{a}_{\perp}\) is the vector rejection of \(\vec{a}\) from \(\vec{b}\) (the orthogonal component).

The vector projection is calculated using the dot product:

\[\text{proj}_{\vec{b}}(\vec{a}) = \frac{\vec{a} \cdot \vec{b}}{\vec{b} \cdot \vec{b}} \vec{b}\]

Subtracting the projection from the original vector yields the vector rejection:

\[\text{rej}_{\vec{b}}(\vec{a}) = \vec{a} - \text{proj}_{\vec{b}}(\vec{a}) = \vec{a} - \left( \frac{\vec{a} \cdot \vec{b}}{\vec{b} \cdot \vec{b}} \right) \vec{b}\]

GLSL Implementation for Arbitrary Vectors

When the reference vector b is not guaranteed to be normalized, use the GLSL dot() function to handle both the numerator and denominator:

vec3 vectorRejection(vec3 a, vec3 b) {
    float bDotB = dot(b, b);
    
    // Prevent division by zero if b is a null vector
    if (bDotB == 0.0) {
        return a;
    }
    
    vec3 projection = (dot(a, b) / bDotB) * b;
    return a - projection;
}

Optimized Implementation for Normalized Vectors

In shader programming, reference vectors (such as surface normals, light directions, or camera axes) are frequently normalized beforehand. When the reference vector b has a length of 1, the term \(\vec{b} \cdot \vec{b}\) evaluates to 1.0, eliminating the need for division:

vec3 vectorRejectionUnit(vec3 a, vec3 unitB) {
    vec3 projection = dot(a, unitB) * unitB;
    return a - projection;
}

Performance Consideration

In real-time graphics pipelines, avoid dynamic branching by ensuring the reference vector is non-zero before passing it to the function. Pre-normalizing reference vectors outside the shader or in the vertex stage allows you to use the optimized multiplication-only variant, reducing instruction cycles in the fragment shader.