Transforming Tangent Space Normal Maps in GLSL

This article explains how tangent, bitangent, and normal vectors form the TBN matrix to transform tangent-space normal maps into world or view space within OpenGL Shading Language (GLSL). Normal maps store surface perturbations relative to an arbitrary 2D texture plane, requiring a dynamic coordinate transformation to interact correctly with scene lighting. By understanding the geometric roles of the tangent, bitangent, and normal vectors, you will learn how to construct the TBN basis matrix, sample normal maps, and apply the transformations directly inside your vertex and fragment shaders.

Understanding Tangent Space

A normal map encodes 3D vector directions as RGB color values. In a standard tangent-space normal map:

  • The Red channel corresponds to the X-axis (left/right).
  • The Green channel corresponds to the Y-axis (up/down).
  • The Blue channel corresponds to the Z-axis (pointing directly outwards from the surface).

Because these vectors point predominantly along the positive Z-axis, tangent-space normal maps appear predominantly blue. However, these vectors are local to the 2D surface of the geometry. To calculate lighting, the renderer must translate these local offsets into the coordinate space where lighting occurs—typically world space or view (camera) space.

The Role of Tangent, Bitangent, and Normal Vectors

To align the normal map with the 3D surface, a local coordinate frame is defined for each vertex using three mutually perpendicular unit vectors:

  1. Normal (\(N\)): The vector perpendicular to the triangle's surface.
  2. Tangent (\(T\)): The vector lying within the surface plane, aligned with the positive horizontal (\(U\)) axis of the texture coordinates.
  3. Bitangent (\(B\)): The vector lying within the surface plane, aligned with the positive vertical (\(V\)) axis of the texture coordinates.

Together, these three vectors form an orthonormal basis known as the TBN matrix. The bitangent can be supplied per-vertex or derived mathematically in the vertex shader by computing the cross product of the normal and the tangent, often multiplied by a scalar to account for UV mirroring:

\[\mathbf{B} = (\mathbf{N} \times \mathbf{T}) \cdot \text{handedness}\]

Constructing the TBN Matrix in GLSL

Transforming vectors between spaces can be done in two ways: transforming light and view vectors into tangent space within the vertex shader, or transforming the sampled normal into world/view space within the fragment shader. The latter is preferred in modern deferred and physically based rendering pipelines.

1. Vertex Shader Implementation

In the vertex shader, the incoming attributes for normal and tangent are transformed by the model or normal matrix. The Gram-Schmidt process is typically applied to guarantee that the tangent remains perfectly perpendicular to the normal after interpolation and transformation.

#version 330 core
layout (location = 0) in vec3 aPos;
layout (location = 1) in vec3 aNormal;
layout (location = 2) in vec2 aTexCoords;
layout (location = 3) in vec4 aTangent; // xyz: tangent, w: handedness

out vec2 TexCoords;
out mat3 vTBN;

uniform mat4 model;
uniform mat3 normalMatrix;

void main()
{
    TexCoords = aTexCoords;

    // Transform vectors to world space
    vec3 N = normalize(normalMatrix * aNormal);
    vec3 T = normalize(normalMatrix * aTangent.xyz);

    // Gram-Schmidt re-orthogonalization
    T = normalize(T - dot(T, N) * N);

    // Calculate bitangent
    vec3 B = cross(N, T) * aTangent.w;

    // Construct the TBN matrix
    vTBN = mat3(T, B, N);

    gl_Position = model * vec4(aPos, 1.0);
}

2. Fragment Shader Implementation

In the fragment shader, the color from the normal map is read and converted from the normalized \([0, 1]\) color range back to the \([-1, 1]\) vector range. The unpacked vector is then multiplied by the interpolated TBN matrix to rotate it into world space.

#version 330 core
out vec4 FragColor;

in vec2 TexCoords;
in mat3 vTBN;

uniform sampler2D normalMap;

void main()
{
    // Sample the normal map (values in range [0, 1])
    vec3 normal = texture(normalMap, TexCoords).rgb;

    // Unpack the normal to range [-1, 1]
    normal = normal * 2.0 - 1.0;

    // Transform normal to world space and normalize
    vec3 worldNormal = normalize(vTBN * normal);

    // Use worldNormal in lighting calculations
    FragColor = vec4(worldNormal * 0.5 + 0.5, 1.0);
}

Deriving the TBN Matrix in the Fragment Shader

If tangent attributes are unavailable in the mesh data, the TBN basis can be derived dynamically in the fragment shader using screen-space derivatives (dFdx and dFdy). By evaluating the rate of change of world-space positions relative to the rate of change of texture coordinates, GLSL can approximate the surface tangent and bitangent at runtime:

vec3 calculateNormal(vec3 worldPos, vec3 worldNormal, vec2 texCoords, sampler2D map)
{
    vec3 dPdx = dFdx(worldPos);
    vec3 dPdy = dFdy(worldPos);
    vec2 dUvdx = dFdx(texCoords);
    vec2 dUvdy = dFdy(texCoords);

    vec3 N = normalize(worldNormal);
    vec3 T = normalize(dPdx * dUvdy.y - dPdy * dUvdx.y);
    vec3 B = -normalize(cross(N, T));
    mat3 TBN = mat3(T, B, N);

    vec3 mapNormal = texture(map, texCoords).rgb * 2.0 - 1.0;
    return normalize(TBN * mapNormal);
}

By leveraging the TBN matrix, GLSL allows 2D textures to define micro-surface details without altering the underlying geometric complexity, yielding high visual fidelity during dynamic lighting passes.