Parallax Occlusion Mapping in Custom glTF Pipelines

Parallax occlusion mapping (POM) enhances surface depth in real-time rendering by raymarching height maps within tangent space to simulate complex geometric detail without adding polygons. Integrating POM into custom glTF shader pipelines requires bridging glTF 2.0's standard metallic-roughness physically based rendering (PBR) specification with custom shader extensions. This process involves packing height data into glTF texture slots, transforming camera vectors to tangent space, performing ray intersection algorithms in the fragment shader, and updating texture coordinate offsets prior to standard PBR lighting evaluations.

glTF Specification Constraints and Material Overrides

The core glTF 2.0 specification natively supports normal maps, base color, metallic-roughness, emissive, and ambient occlusion, but it omits native displacement or height maps. To introduce POM, you must implement a custom pipeline integration using one of two methods:

glTF Extensions: Define a vendor or application-level extension (such as KHR_materials_displacement or a custom EXT_ prefix) to store height map references, scale factors, and minimum/maximum raymarching steps directly in the asset JSON.
Shader Injection: Intercept the material loading process in runtime engines (such as Three.js, Babylon.js, or custom WebGPU/Vulkan engines) and swap the standard glTF PBR fragment stage with a custom uber-shader that expects height data.

Texture Packing and Tangent Space Setup

POM requires an accurate height map alongside the glTF mesh's tangent space vectors. To optimize texture fetches, height maps should be packed into available texture channels. A common standard is storing the height data in the alpha channel of the glTF normal map texture (RGB for normal, A for height).

glTF models utilizing normal mapping typically provide explicit four-component tangent attributes (vec4 TANGENT), where the w component indicates handedness. In the vertex shader, calculate the Tangent-Bitangent-Normal (TBN) matrix:

vec3 normal = normalize(u_NormalMatrix * a_Normal);
vec3 tangent = normalize(u_NormalMatrix * a_Tangent.xyz);
vec3 bitangent = cross(normal, tangent) * a_Tangent.w;
mat3 tbnMatrix = mat3(tangent, bitangent, normal);

// Transform view direction from view/world space to tangent space
v_TangentViewDir = transpose(tbnMatrix) * (u_CameraPosition - v_WorldPosition);

Fragment Shader Raymarching Implementation

In the fragment shader, POM executes before any lighting calculations. It determines the point where the tangent-space view ray intersects the displacement profile.

Layer Slicing: Determine the number of ray steps dynamically based on the viewing angle. Grazing angles require more steps to avoid stepping artifacts:
```
float numLayers = mix(maxLayers, minLayers, abs(dot(vec3(0.0, 0.0, 1.0), normalize(v_TangentViewDir))));
```
Raymarching: Step along the view vector in tangent space, tracking the current layer depth and UV offset until the sampled height exceeds the current ray depth.
Occlusion Interpolation: After finding the intersection layer, linearly interpolate between the depth coordinates before and after the collision point to achieve smooth surface transitions without geometric stair-stepping.

vec2 ParallaxMapping(vec2 texCoords, vec3 viewDir) {
    float layerDepth = 1.0 / numLayers;
    float currentLayerDepth = 0.0;
    vec2 P = viewDir.xy / viewDir.z * u_HeightScale; 
    vec2 deltaTexCoords = P / numLayers;
    
    vec2 currentTexCoords = texCoords;
    float currentDepthMapValue = 1.0 - texture(u_HeightMap, currentTexCoords).r;
    
    while(currentLayerDepth < currentDepthMapValue) {
        currentTexCoords -= deltaTexCoords;
        currentDepthMapValue = 1.0 - texture(u_HeightMap, currentTexCoords).r;
        currentLayerDepth += layerDepth;
    }
    
    vec2 prevTexCoords = currentTexCoords + deltaTexCoords;
    float afterDepth  = currentDepthMapValue - currentLayerDepth;
    float beforeDepth = (1.0 - texture(u_HeightMap, prevTexCoords).r) - currentLayerDepth + layerDepth;
    
    float weight = afterDepth / (afterDepth - beforeDepth);
    return prevTexCoords * weight + currentTexCoords * (1.0 - weight);
}

Feeding Displaced Coordinates into the glTF PBR Pipeline

Once the modified UV coordinates are calculated, pass them into every subsequent glTF texture sampling function. This includes the base color (pbrMetallicRoughness.baseColorTexture), normal map, metallic-roughness texture, and ambient occlusion map.

Standard texture derivative functions (dFdx, dFdy) can cause visual artifacts at silhouette edges or texture boundaries due to the non-linear coordinate jumps introduced by POM. To resolve this, compute the screen-space derivatives of the original UVs prior to raymarching and use explicit derivative sampling (textureGrad in GLSL) across all glTF material texture lookups:

vec2 dx = dFdx(v_TexCoords);
vec2 dy = dFdy(v_TexCoords);

vec2 displacedUV = ParallaxMapping(v_TexCoords, normalize(v_TangentViewDir));

// Sample all standard glTF textures using the displaced UVs and original derivatives
vec4 baseColor = textureGrad(u_BaseColorTexture, displacedUV, dx, dy);
vec4 normalData = textureGrad(u_NormalTexture, displacedUV, dx, dy);
vec4 metallicRoughness = textureGrad(u_MetallicRoughnessTexture, displacedUV, dx, dy);

Self-Shadowing and Quality Optimizations

To maximize the visual realism of the integration, compute self-shadowing by casting a secondary ray from the displaced surface point toward the light source in tangent space. If the ray encounters height map values greater than its current trajectory height, apply an occlusion factor to the glTF light computation. Finally, discard pixels whose displaced UV coordinates fall outside the standard [0.0, 1.0] range if texture tiling is disabled to prevent edge clamping artifacts along the borders of meshes.