How Does Cross Product Work on vec3 in GLSL?

In the OpenGL Shading Language (GLSL), the cross() function calculates the geometric cross product of two three-dimensional floating-point vectors (vec3), returning a new vec3 that is perpendicular to both inputs following the right-hand rule. This operation is fundamental to 3D rendering pipelines, serving as the mathematical backbone for computing surface normals, building camera coordinate frames, and generating tangent spaces. Understanding how GLSL evaluates cross(vec3 x, vec3 y), along with its mathematical formula, performance considerations, and edge cases, is essential for writing accurate vertex and fragment shaders.

Mathematical Definition in GLSL

The built-in cross(x, y) function accepts two parameters of type vec3 (or precision-qualified variants like highp vec3) and evaluates the standard algebraic cross product definition:

\[\mathbf{x} \times \mathbf{y} = \begin{pmatrix} x_y y_z - x_z y_y \\ x_z y_x - x_x y_z \\ x_x y_y - x_y y_x \end{pmatrix}\]

Expressed in GLSL component swizzling and arithmetic, the operation behaves identically to the following manual implementation:

vec3 manualCross(vec3 x, vec3 y) {
    return vec3(
        x.y * y.z - x.z * y.y,
        x.z * y.x - x.x * y.z,
        x.x * y.y - x.y * y.x
    );
}

GPU hardware natively optimizes this operation into efficient multiply-subtract (or fused multiply-add) instruction pairs with negligible execution cost.

Core Applications in Shaders

1. Generating Surface Normals

When vertex normals are unavailable, a fragment shader can compute a flat facet normal dynamically using screen-space partial derivatives (dFdx and dFdy) combined with cross():

vec3 worldPos = v_worldPosition;
vec3 dX = dFdx(worldPos);
vec3 dY = dFdy(worldPos);
vec3 surfaceNormal = normalize(cross(dX, dY));

2. Constructing Orthogonal Basis Vectors

Generating orthonormal coordinate frames—such as Tangent-Bitangent-Normal (TBN) matrices for normal mapping or view-matrix construction—relies on finding mutually orthogonal axes:

vec3 normal = normalize(v_normal);
vec3 tangent = normalize(v_tangent);
vec3 bitangent = cross(normal, tangent) * v_tangentSign;
mat3 TBN = mat3(tangent, bitangent, normal);

Key Considerations and Edge Cases

Output Magnitude: The magnitude of the resulting vector equals \(\Vert{}\mathbf{x}\Vert{} \Vert{}\mathbf{y}\Vert{} \sin(\theta)\), where \(\theta\) is the angle between \(\mathbf{x}\) and \(\mathbf{y}\). Even if both input vectors are unit length, the output vector is only unit length when the inputs are strictly perpendicular (\(\sin(90^\circ) = 1\)). To use the result as a directional vector or normal, wrap the result in normalize().
Collinear Vectors: If vectors \(\mathbf{x}\) and \(\mathbf{y}\) are parallel or anti-parallel (\(\theta = 0^\circ\) or \(180^\circ\)), cross(x, y) evaluates to vec3(0.0, 0.0, 0.0). Passing a zero vector to normalize() results in undefined behavior or NaN values depending on the driver implementation.
Handedness and Order of Operands: The cross product is anticommutative: cross(x, y) == -cross(y, x). Reversing the argument order flips the vector direction by 180 degrees, which can invert lighting calculations or cause backface-shading artifacts.