How Does Cross Product Work on vec3 in GLSL?

In the OpenGL Shading Language (GLSL), the cross() function calculates the geometric cross product of two three-dimensional floating-point vectors (vec3), returning a new vec3 that is perpendicular to both inputs following the right-hand rule. This operation is fundamental to 3D rendering pipelines, serving as the mathematical backbone for computing surface normals, building camera coordinate frames, and generating tangent spaces. Understanding how GLSL evaluates cross(vec3 x, vec3 y), along with its mathematical formula, performance considerations, and edge cases, is essential for writing accurate vertex and fragment shaders.

Mathematical Definition in GLSL

The built-in cross(x, y) function accepts two parameters of type vec3 (or precision-qualified variants like highp vec3) and evaluates the standard algebraic cross product definition:

\[\mathbf{x} \times \mathbf{y} = \begin{pmatrix} x_y y_z - x_z y_y \\ x_z y_x - x_x y_z \\ x_x y_y - x_y y_x \end{pmatrix}\]

Expressed in GLSL component swizzling and arithmetic, the operation behaves identically to the following manual implementation:

vec3 manualCross(vec3 x, vec3 y) {
    return vec3(
        x.y * y.z - x.z * y.y,
        x.z * y.x - x.x * y.z,
        x.x * y.y - x.y * y.x
    );
}

GPU hardware natively optimizes this operation into efficient multiply-subtract (or fused multiply-add) instruction pairs with negligible execution cost.

Core Applications in Shaders

1. Generating Surface Normals

When vertex normals are unavailable, a fragment shader can compute a flat facet normal dynamically using screen-space partial derivatives (dFdx and dFdy) combined with cross():

vec3 worldPos = v_worldPosition;
vec3 dX = dFdx(worldPos);
vec3 dY = dFdy(worldPos);
vec3 surfaceNormal = normalize(cross(dX, dY));

2. Constructing Orthogonal Basis Vectors

Generating orthonormal coordinate frames—such as Tangent-Bitangent-Normal (TBN) matrices for normal mapping or view-matrix construction—relies on finding mutually orthogonal axes:

vec3 normal = normalize(v_normal);
vec3 tangent = normalize(v_tangent);
vec3 bitangent = cross(normal, tangent) * v_tangentSign;
mat3 TBN = mat3(tangent, bitangent, normal);

Key Considerations and Edge Cases