Transform Normals with gl_ObjectToWorldEXT in GLSL

This article explains how to correctly transform surface normal vectors from object space to world space inside GLSL closest hit shaders using Vulkan ray tracing extensions. It covers the mathematical requirements for transforming directional vectors under non-uniform scaling, demonstrates the implementation using gl_ObjectToWorldEXT, and highlights performance optimizations utilizing the companion matrix gl_WorldToObjectEXT.

The Mathematics of Normal Transformation

Surface normals represent directions perpendicular to a surface rather than spatial coordinates. While position coordinates can be directly multiplied by an affine model-to-world transformation matrix (\(M\)), transforming normal vectors with the same matrix breaks perpendicularity whenever non-uniform scaling or shearing is present.

To preserve the orthogonal relationship between surface tangent vectors (\(T\)) and normal vectors (\(N\)) such that \(T \cdot N = 0\), normals must be transformed using the inverse transpose of the 3x3 linear portion of the transformation matrix:

\[N_{world} = (M^{-1})^T \cdot N_{object}\]

Using gl_ObjectToWorldEXT Directly

In GLSL ray tracing shaders (.rchit), the affine transformation matrix from object space to world space is provided by the built-in variable:

// Built-in declaration
// mat4x3 gl_ObjectToWorldEXT;

Because gl_ObjectToWorldEXT has the type mat4x3, it contains a 3x3 rotation-and-scale submatrix along with a translation column. To transform an object-space normal using gl_ObjectToWorldEXT directly:

  1. Extract the upper 3x3 rotation and scaling components.
  2. Invert the 3x3 matrix.
  3. Transpose the inverted matrix.
  4. Multiply by the object-space normal and normalize the result.
// Extract the 3x3 linear transformation component
mat3 objectToWorld3x3 = mat3(gl_ObjectToWorldEXT);

// Compute the normal matrix (inverse transpose)
mat3 normalMatrix = transpose(inverse(objectToWorld3x3));

// Transform and re-normalize the normal
vec3 worldNormal = normalize(normalMatrix * objectNormal);

While mathematically complete, evaluating inverse() inside the closest hit shader introduces a runtime performance overhead for every ray intersection.

The Optimal Approach: Pairing with gl_WorldToObjectEXT

Vulkan ray tracing pipelines provide the world-to-object matrix via gl_WorldToObjectEXT, which is precomputed by the driver as the inverse of gl_ObjectToWorldEXT:

\[\text{mat3}(gl\_WorldToObjectEXT) \approx M^{-1}\]

Because the inverse matrix is already available, the explicit inverse() call can be omitted entirely. Transposing gl_WorldToObjectEXT produces the exact inverse-transpose matrix required:

// Extract the 3x3 component of the inverse matrix
mat3 worldToObject3x3 = mat3(gl_WorldToObjectEXT);

// Transpose the inverse directly (Inverse Transpose)
vec3 worldNormal = normalize(transpose(worldToObject3x3) * objectNormal);

Alternatively, utilizing GLSL's row-vector by matrix multiplication order performs an implicit transpose:

// Equivalent to transpose(mat3(gl_WorldToObjectEXT)) * objectNormal
vec3 worldNormal = normalize(objectNormal * mat3(gl_WorldToObjectEXT));

Implementation in a Closest Hit Shader

Below is an integrated example inside a closest hit shader (.rchit):

#version 460
#extension GL_EXT_ray_tracing : require

layout(location = 0) rayPayloadInEXT vec3 hitColor;
hitAttributeEXT vec2 attribs;

void main()
{
    // Retrieve or calculate object-space geometric normal
    vec3 objectNormal = vec3(0.0, 1.0, 0.0);

    // Transform to world space via inverse-transpose logic
    // Using gl_WorldToObjectEXT avoids computing inverse(mat3(gl_ObjectToWorldEXT))
    vec3 worldNormal = normalize(objectNormal * mat3(gl_WorldToObjectEXT));

    // Proceed with lighting calculations using worldNormal
    vec3 lightDir = normalize(vec3(1.0, 2.0, -1.0));
    float diffuse = max(dot(worldNormal, lightDir), 0.0);
    
    hitColor = vec3(diffuse);
}

Key Considerations

  • Normalization: Transforming a normal vector alters its length whenever scaling is applied to the instance. Always wrap the transformed vector in normalize().
  • Uniform Scaling: If an instance strictly uses uniform scaling and rigid rotations, the inverse transpose is equivalent to the original matrix up to a scalar factor. In such constrained pipelines, normalize(mat3(gl_ObjectToWorldEXT) * objectNormal) is sufficient.
  • Affine Layout: Remember that both gl_ObjectToWorldEXT and gl_WorldToObjectEXT are mat4x3 matrices (4 columns, 3 rows in GLSL conventions), meaning slicing via mat3(...) extracts the primary linear transformation while dropping translation components.