Hemi-Octahedral Normal Compression in GLSL

Hemi-octahedral normal vector compression is an efficient encoding technique designed for 3D unit vectors constrained to a hemisphere, such as tangent-space surface normals where \(z \ge 0\). By projecting the hemisphere onto the surface of an octahedron and unfolding it onto a two-dimensional grid, this method maximizes precision across two-component texture formats (such as 8-bit or 16-bit SNORM/UNORM). This article provides a concise technical breakdown of the hemi-octahedral projection and presents optimized GLSL implementations for both encoding and decoding hemispherical normals.

The Hemi-Octahedral Projection

Standard octahedral mapping projects a unit sphere onto an octahedron using the \(L_1\) norm (\(\|v\|_1 = |x| + |y| + |z|\)) and folds the negative hemisphere to fit within a \([-1, 1]^2\) square domain.

When vectors are known to lie strictly within a hemisphere (\(z \ge 0\)), the folding step for negative \(z\) is unnecessary. Instead, projecting the upper hemisphere onto the octahedron results in a diamond shape defined by \(|x| + |y| \le 1\). To optimize texture coordinate utilization and precision, this diamond is rotated by 45 degrees and scaled, perfectly fitting it into the square \([-1, 1]^2\) domain without wasting any data range.

GLSL Encoding Implementation

To encode a 3D normal vector \((v_x, v_y, v_z)\) where \(z \ge 0\):

  1. Project the vector onto the octahedron by dividing by the \(L_1\) norm: \((|x| + |y| + z)\).
  2. Rotate the resulting 2D coordinates \((p_x, p_y)\) by 45 degrees and scale by \(\sqrt{2}\) using the transformation: \[e_x = p_x + p_y\] \[e_y = p_x - p_y\]
// Encodes a unit vector with z >= 0 into [-1, 1]^2
vec2 encodeHemiOctahedral(vec3 n) {
    // Project onto the octahedron (L1 norm)
    vec2 p = n.xy / (abs(n.x) + abs(n.y) + n.z);
    
    // Rotate and scale diamond to fill the [-1, 1] square
    return vec2(p.x + p.y, p.x - p.y);
}

If storing in an unsigned normalized format (UNORM), transform the output from \([-1, 1]\) to \([0, 1]\):

vec2 encodeHemiOctahedralUNORM(vec3 n) {
    vec2 enc = encodeHemiOctahedral(n);
    return enc * 0.5 + 0.5;
}

GLSL Decoding Implementation

Decoding reverses the coordinate rotation and computes the reconstructed \(z\) component before normalizing the vector:

  1. Invert the 45-degree rotation and scale: \[p_x = \frac{e_x + e_y}{2}\] \[p_y = \frac{e_x - e_y}{2}\]
  2. Reconstruct \(z\) using the \(L_1\) surface definition: \[z = 1.0 - |p_x| - |p_y|\]
  3. Normalize the resulting vector \((p_x, p_y, z)\) to project it back onto the unit sphere.
// Decodes a [-1, 1]^2 encoded vector back into a unit vec3 (z >= 0)
vec3 decodeHemiOctahedral(vec2 e) {
    // Reverse the 45-degree rotation and scaling
    vec2 p = vec2(e.x + e.y, e.x - e.y) * 0.5;
    
    // Reconstruct z from the octahedral surface
    float z = 1.0 - abs(p.x) - abs(p.y);
    
    // Renormalize back to the unit sphere
    return normalize(vec3(p.x, p.y, z));
}

If reading from an unsigned texture format:

vec3 decodeHemiOctahedralUNORM(vec2 enc) {
    vec2 e = enc * 2.0 - 1.0;
    return decodeHemiOctahedral(e);
}

Considerations and Precision

  • Uniform Precision: Unlike spherical coordinates (azimuth/elevation), hemi-octahedral mapping maintains nearly uniform area distribution and avoids singularity issues at the poles.
  • Storage Formats: Storing the encoded result in an RG16_SNORM texture yields virtually lossless reconstruction for normal mapping. Using an 8-bit RG8_UNORM format offers higher visual quality compared to dropping the \(z\) channel directly (\(z = \sqrt{1 - x^2 - y^2}\)), which suffers from high quantization error near glancing angles.
  • Domain Restrictions: This implementation strictly requires \(z \ge 0\). If arbitrary vectors across the entire sphere need to be encoded, standard full-sphere octahedral mapping with folding logic must be used instead.