Octahedral Normal Encoding and Decoding in GLSL

Octahedral unit vector encoding is an efficient mathematical technique used in computer graphics to project a 3D unit normal vector onto a 2D surface, and vice versa. By mapping a sphere onto an octahedron and then unfolding it onto an axis-aligned square domain \([-1, 1]^2\), this method provides a near-optimal distribution of precision with minimal distortion. This article covers the exact mathematical formulation of octahedral vector encoding and decoding, followed by its complete, branch-free implementation in GLSL.

Mathematical Formulation

The transformation relies on the \(L_1\) norm (Manhattan norm) to project a unit vector onto the surface of a regular octahedron, followed by folding operations to unwrap the lower hemisphere onto the corner triangles of a 2D square.

Encoding (3D to 2D)

Given a 3D unit vector \(\mathbf{n} = (x, y, z)\) such that \(\|\mathbf{n}\|_2 = 1\):

  1. Octahedral Projection: Project the vector onto the surface of an octahedron by dividing by its \(L_1\) norm: \[p = \frac{\mathbf{n}}{|x| + |y| + |z|}\]

  2. Octahedron Unfolding:

    • If \(z \ge 0\), the vector lies in the upper hemisphere. The encoded coordinates are simply the \(x\) and \(y\) components of the projection: \[(u, v) = (p_x, p_y)\]
    • If \(z < 0\), the vector lies in the lower hemisphere. The corners must be folded over the diagonals according to the signs of \(p_x\) and \(p_y\): \[u = (1 - |p_y|) \cdot \text{sgn}(p_x)\] \[v = (1 - |p_x|) \cdot \text{sgn}(p_y)\]

The resulting coordinates \((u, v)\) reside entirely within the range \([-1, 1]\).

Decoding (2D to 3D)

Given an encoded 2D coordinate \(\mathbf{e} = (u, v) \in [-1, 1]^2\):

  1. Initial Plane Reconstruction: Reconstruct the \(z\)-coordinate using the octahedral constraint: \[z = 1 - (|u| + |v|)\]

  2. Invert the Fold:

    • If \(z < 0\), the vector originated from the lower hemisphere. Mirror the \(x\) and \(y\) coordinates back into their original quadrants: \[x = (1 - |v|) \cdot \text{sgn}(u)\] \[y = (1 - |u|) \cdot \text{sgn}(v)\]
    • If \(z \ge 0\), retain \(x = u\) and \(y = v\).

  3. Renormalization: Project the resulting vector \((x, y, z)\) back onto the unit sphere using the Euclidean norm: \[\mathbf{n} = \frac{(x, y, z)}{\sqrt{x^2 + y^2 + z^2}}\]

GLSL Implementation

Standard GLSL sign() returns 0.0 when the input is zero, which introduces boundary artifacts along coordinate axes. A robust implementation requires a modified sign function that maps zero to 1.0.

// Custom sign function to avoid zero-value discontinuities
vec2 octSign(vec2 v) {
    return vec2(
        v.x >= 0.0 ? 1.0 : -1.0,
        v.y >= 0.0 ? 1.0 : -1.0
    );
}

// Encodes a normalized 3D vector to a 2D vector in [-1, 1]^2
vec2 octEncode(vec3 n) {
    // Project to the octahedral plane (L1 norm)
    vec2 p = n.xy * (1.0 / (abs(n.x) + abs(n.y) + abs(n.z)));
    
    // Fold the lower hemisphere
    return (n.z < 0.0) ? ((1.0 - abs(p.yx)) * octSign(p)) : p;
}

// Decodes a 2D vector in [-1, 1]^2 back to a normalized 3D vector
vec3 octDecode(vec2 e) {
    vec3 n = vec3(e.xy, 1.0 - abs(e.x) - abs(e.y));
    
    // Unfold lower hemisphere
    float t = max(-n.z, 0.0);
    n.xy += vec2(n.x >= 0.0 ? -t : t, n.y >= 0.0 ? -t : t);
    
    return normalize(n);
}

Quantization for Storage

To store the encoded normal in an unsigned 16-bit or 32-bit format (such as RGBA8_UNORM or RG16_SNORM):

  • For signed formats (SNORM), the encoded vector vec2 can be stored directly.
  • For unsigned formats (UNORM), remap the values from \([-1, 1]\) to \([0, 1]\) using \(0.5 \cdot \mathbf{e} + 0.5\) during encoding, and invert with \(2.0 \cdot \mathbf{e}_{unorm} - 1.0\) prior to decoding.