gl_ObjectToWorldEXT Normal Transformation in GLSL

Transforming surface normals from object space to world space in GLSL ray tracing shaders requires understanding how the built-in gl_ObjectToWorldEXT matrix operates. While positions and standard direction vectors use forward affine transforms, normal vectors require the inverse-transpose of the transformation matrix to preserve orthogonality under non-uniform scaling. This article explains the exact mathematical operation performed by gl_ObjectToWorldEXT, why direct multiplication can fail, and how to correctly transform normals in hardware ray tracing pipelines.

The Role of gl_ObjectToWorldEXT

In GLSL ray tracing extensions (such as GL_EXT_ray_tracing), gl_ObjectToWorldEXT is a built-in uniform matrix of type mat4x3. It represents the 4x3 affine transformation matrix responsible for converting coordinates from the current instance's object space to world space.

When multiplied directly with a direction vector using its upper-left \(3\times3\) components, it applies standard linear transformations: rotation, reflection, and scaling:

\[\mathbf{v}_{\text{world}} = \mathbf{M}_{3\times3} \cdot \mathbf{v}_{\text{object}}\]

vec3 worldDirection = mat3(gl_ObjectToWorldEXT) * objectDirection;

Why Direct Transformation Fails for Normals

Surface normals represent planes tangent to the geometry rather than displacement vectors. If a 3D model undergoes non-uniform scaling (e.g., stretched along the X-axis by \(2.0\)), applying the forward transformation matrix directly to the normal vector distorts its angle, causing it to no longer be perpendicular to the surface.

To preserve perpendicularity, normal vectors must be transformed by the inverse transpose of the upper-left \(3\times3\) portion of the object-to-world matrix:

\[\mathbf{n}_{\text{world}} = (\mathbf{M}_{3\times3}^{-1})^{T} \cdot \mathbf{n}_{\text{object}}\]

If you rely solely on gl_ObjectToWorldEXT to compute this transformation, you must compute the inverse transpose explicitly:

mat3 normalMatrix = transpose(inverse(mat3(gl_ObjectToWorldEXT)));
vec3 worldNormal = normalize(normalMatrix * objectNormal);

While mathematically correct, calling inverse() in a real-time shader introduces unnecessary computational overhead.

The Optimal Transformation Using gl_WorldToObjectEXT

Vulkan and GLSL ray tracing APIs provide the precomputed inverse matrix through gl_WorldToObjectEXT. Because gl_WorldToObjectEXT represents \(\mathbf{M}^{-1}\), you only need to transpose its upper-left \(3\times3\) submatrix to obtain the correct normal transformation matrix:

\[(\mathbf{M}^{-1})^{T} = (\text{mat3}(\text{gl\_WorldToObjectEXT}))^{T}\]

In GLSL, multiplying a vector on the left side of a matrix evaluates to the same result as multiplying the transposed matrix on the right side of the vector. Therefore, the transformation can be performed efficiently without calling transpose() or inverse():

// Multiplies the normal by the transpose of the world-to-object matrix
vec3 worldNormal = normalize(objectNormal * mat3(gl_WorldToObjectEXT));

Alternatively, writing the transpose explicitly produces the identical assembly result:

vec3 worldNormal = normalize(transpose(mat3(gl_WorldToObjectEXT)) * objectNormal);

Summary of Behavior

  • Direct use of gl_ObjectToWorldEXT: Applies standard forward rotation and scale. This is only valid for normals if the instance transform is purely rigid (rotation and translation only) or uses uniform scaling.
  • Non-uniform scaling: Requires the inverse-transpose transformation. Rather than computing transpose(inverse(mat3(gl_ObjectToWorldEXT))), use gl_WorldToObjectEXT transposed to achieve the correct mathematical result efficiently.
  • Normalization: Any normal transformed into world space must be passed through normalize(), as scaling operations alter the vector's unit length.