GPU Texture Matrix Palettes for glTF Crowd Rendering

Rendering large crowds of animated glTF characters in real time introduces significant performance bottlenecks, primarily due to uniform buffer limits and excessive draw calls. Storing skeletal joint transformation matrices inside floating-point GPU textures—often referred to as texture-based matrix palettes—bypasses standard uniform buffer limits and enables massive hardware instancing across thousands of animated agents in a single draw call. This article details how to pack transformation matrices into textures, fetch them within vertex shaders, and implement animation baking to maximize glTF crowd rendering performance.

The Bottleneck of Standard Skinning

Standard glTF skeletal animation binds joint matrices via Uniform Buffer Objects (UBOs) or uniform arrays. Graphics drivers impose strict size limits on UBOs (typically 64 KB), which restricts the maximum number of simultaneous joints per draw call. While a single character with 64 joints fits comfortably, rendering hundreds of unique animated instances requires either separate draw calls per instance or constant uniform buffer updates, heavily taxing the CPU and graphics API pipeline.

Texture-Based Palette Architecture

A texture-based matrix palette offloads the entire skinning state to GPU memory by encoding \(4 \times 4\) or \(4 \times 3\) transformation matrices into texels.

  • Pixel Format: Use a 32-bit floating-point format, typically RGBA32F. Each pixel contains four float values \((x, y, z, w)\).
  • Matrix Representation: A full \(4 \times 4\) affine matrix requires four consecutive pixels. Because the last row of an affine transformation matrix in 3D space is always \([0, 0, 0, 1]\), you can discard it and store only a \(4 \times 3\) matrix. This reduces memory footprint and bandwidth by 25%, packing a joint into three pixels:
    • Pixel 0: Row 0 \((m_{00}, m_{01}, m_{02}, m_{03})\)
    • Pixel 1: Row 1 \((m_{10}, m_{11}, m_{12}, m_{13})\)
    • Pixel 2: Row 2 \((m_{20}, m_{21}, m_{22}, m_{23})\)
  • Texture Layout: A common structure is a 2D texture where the horizontal axis corresponds to the joint index (multiplied by the number of pixels per matrix), and the vertical axis corresponds to either the instance ID or pre-baked animation frame index.

Pre-Baking vs. Dynamic Compute Updates

Depending on your application, joint textures can be populated dynamically or pre-baked:

  1. Pre-Baked Animations: Ideal for background crowds. All keyframe poses of glTF animation clips are sampled offline or at load time and written sequentially into the texture. Each row represents a specific animation frame. At runtime, the CPU simply passes an animation time or frame offset per instance, requiring zero skeletal math during frame execution.
  2. Compute Shader Updates: Ideal for dynamic crowd behaviors (such as ragdolls or procedural IK). The CPU sends minimal transform data, and a GPU compute shader evaluates the glTF node hierarchy, computes inverse bind matrix multiplications, and writes the results directly into a writable storage texture (image2D).

Fetching Matrices in the Vertex Shader

glTF meshes specify skinning data using JOINTS_0 (joint indices) and WEIGHTS_0 (joint weights). When using a texture matrix palette, the vertex shader reconstructs the final transformation matrix using integer texel lookups via texelFetch.

Below is a GLSL implementation using \(4 \times 3\) matrices:

#version 450 core

layout(location = 0) in vec3 a_Position;
layout(location = 1) in vec3 a_Normal;
layout(location = 2) in uvec4 a_Joints;
layout(location = 3) in vec4 a_Weights;

// Instanced attributes
layout(location = 4) in int a_FrameIndex; 

uniform sampler2D u_JointMatrixTexture;

mat4 getJointMatrix(uint jointIndex, int frame) {
    int basePixel = int(jointIndex) * 3;
    
    vec4 row0 = texelFetch(u_JointMatrixTexture, ivec2(basePixel + 0, frame), 0);
    vec4 row1 = texelFetch(u_JointMatrixTexture, ivec2(basePixel + 1, frame), 0);
    vec4 row2 = texelFetch(u_JointMatrixTexture, ivec2(basePixel + 2, frame), 0);
    vec4 row3 = vec4(0.0, 0.0, 0.0, 1.0);
    
    return mat4(
        vec4(row0.x, row1.x, row2.x, row3.x),
        vec4(row0.y, row1.y, row2.y, row3.y),
        vec4(row0.z, row1.z, row2.z, row3.z),
        vec4(row0.w, row1.w, row2.w, row3.w)
    );
}

void main() {
    mat4 skinMatrix = 
        a_Weights.x * getJointMatrix(a_Joints.x, a_FrameIndex) +
        a_Weights.y * getJointMatrix(a_Joints.y, a_FrameIndex) +
        a_Weights.z * getJointMatrix(a_Joints.z, a_FrameIndex) +
        a_Weights.w * getJointMatrix(a_Joints.w, a_FrameIndex);

    vec4 skinnedPosition = skinMatrix * vec4(a_Position, 1.0);
    gl_Position = u_ViewProjection * skinnedPosition;
}

Using texelFetch bypasses standard texture filtering and coordinate normalization, pulling raw floating-point data directly by texel coordinates without precision loss or unwanted interpolation.

Implementation Best Practices

  • Texture Arrays for Large Datasets: If the baked animation clips exceed the maximum 2D texture width or height limits of the target hardware (often 8192 or 16384 texels), switch to a 2D Texture Array (sampler2DArray). Assign each clip or character type its own array slice.
  • Dual Quaternion Packing: For further bandwidth savings, represent joint transformations as dual quaternions instead of matrices. A dual quaternion requires only two pixels (8 floats) instead of three or four, lowering fetch overhead and automatically preventing skin-collapsing artifacts during weight blending.
  • Frame Interpolation: If frame rates fluctuate or animations are sampled at low intervals (e.g., 30 Hz on a 120 Hz display), pass two frame indices and a fractional blend factor (lerp) to interpolate between adjacent matrix rows directly in the shader.