Orthonormalize Tangent Basis in GLSL Using Gram-Schmidt

Normal mapping relies on an orthonormal Tangent, Bitangent, and Normal (TBN) frame to accurately transform surface details from tangent space to world or view space. During linear vertex interpolation or mesh deformation, tangent vectors often lose their perpendicularity relative to interpolated normals, causing rendering artifacts. This article details the mathematical formulation and practical GLSL implementation of the Gram-Schmidt process to re-orthogonalize the tangent vector against the surface normal.

The Mathematical Principle

The standard Gram-Schmidt process takes a set of linearly independent vectors and generates an orthogonal basis spanning the same subspace. In computer graphics, the surface normal (\(\mathbf{N}\)) is prioritized as the ground-truth direction for surface orientation. The tangent vector (\(\mathbf{T}\)) must be adjusted so that it lies strictly within the plane perpendicular to \(\mathbf{N}\).

The projection of vector \(\mathbf{T}\) onto unit vector \(\mathbf{N}\) is calculated as:

\[\operatorname{proj}_{\mathbf{N}}(\mathbf{T}) = (\mathbf{T} \cdot \mathbf{N}) \mathbf{N}\]

Subtracting this projection from \(\mathbf{T}\) removes any component pointing along the normal, yielding an orthogonal tangent vector \(\mathbf{T}_{\perp}\):

\[\mathbf{T}_{\perp} = \mathbf{T} - (\mathbf{T} \cdot \mathbf{N}) \mathbf{N}\]

Normalizing \(\mathbf{T}_{\perp}\) produces the unit tangent vector \(\mathbf{T}'\). The bitangent vector (\(\mathbf{B}\)) is then derived using the cross product of \(\mathbf{N}\) and \(\mathbf{T}'\), scaled by the handedness factor (\(w\)) to preserve correct UV coordinate orientation across mirrored seams:

\[\mathbf{B} = (\mathbf{N} \times \mathbf{T}') \cdot w\]

GLSL Implementation

The algorithm can be implemented in either the vertex shader or the fragment shader. Performing it in the fragment shader yields the highest visual accuracy because it accounts for non-linearities introduced by perspective-correct attribute interpolation.

// Inputs:
// inNormal: Interpolated surface normal (vec3)
// inTangent: Interpolated surface tangent with handedness stored in .w (vec4)

// 1. Normalize the surface normal
vec3 N = normalize(inNormal);

// 2. Re-orthogonalize T with respect to N using Gram-Schmidt
vec3 T = normalize(inTangent.xyz - dot(inTangent.xyz, N) * N);

// 3. Compute the bitangent B using the cross product and handedness
vec3 B = cross(N, T) * inTangent.w;

// 4. Construct the orthonormal TBN matrix
mat3 TBN = mat3(T, B, N);

Step-by-Step Breakdown

  1. Normalize the Normal Vector: Vertex attributes lose unit length during rasterization interpolation. Normalizing inNormal first ensures the projection math \((\mathbf{T} \cdot \mathbf{N})\mathbf{N}\) functions correctly without requiring division by \(\|\mathbf{N}\|^2\).
  2. Project and Subtract: The expression dot(inTangent.xyz, N) * N computes the scalar projection of the tangent onto the normal and scales the normal by that factor. Subtracting this from inTangent.xyz guarantees the resulting vector is at a 90-degree angle to N.
  3. Re-normalize the Tangent: Subtracting the projection reduces the magnitude of the tangent vector. A call to normalize() restores it to unit length.
  4. Compute the Orthogonal Bitangent: Calculating cross(N, T) produces a third vector perpendicular to both N and T. Multiplying by inTangent.w preserves texture mirroring where UV coordinates are flipped on symmetric geometry.