Real-Time glTF Lip Sync Using Morph Target Weights

Synchronizing 3D character speech in real time involves converting generated audio into visual mouth movements using standard 3D web formats. Speech synthesis engines achieve real-time glTF lip sync by converting text or generated audio into phonetic timestamps, mapping those units to visual mouth poses called visemes, interpolating morph target weights, and applying these values directly to glTF mesh primitives on each render frame.

1. Acoustic to Phonetic Processing

Modern speech synthesis (Text-to-Speech or TTS) systems generate both an audio stream and associated temporal metadata. During audio synthesis, the engine identifies phonemes—the smallest units of sound—and assigns each an exact start time and duration. Alternatively, raw audio can be processed using acoustic feature extraction (such as Mel-frequency cepstral coefficients or pretrained models like Wav2Vec) to predict speech units frame by frame when phoneme metadata is not provided upfront.

2. Phoneme-to-Viseme Mapping

Because multiple phonemes produce identical mouth shapes, engines map phonemes to a smaller set of visual representations known as visemes. For example, the sounds /p/, /b/, and /m/ all map to a single bilabial closed-mouth viseme. Industry standards like the Microsoft Speech API (SAPI) 21-viseme set, the Disney 12-viseme set, or Apple ARKit’s 52 blendshape standard serve as the intermediate translation layer between acoustic output and geometry deformation.

3. Calculating Morph Target Weights

In the glTF (GL Transmission Format) standard, facial shapes are defined using morph targets (often referred to as blendshapes) stored in a mesh primitive's targets array. Each morph target represents vertex displacements relative to a neutral face mesh.

To drive these targets:

  • Weight Assignment: The engine assigns a scalar weight typically ranging from 0.0 (neutral) to 1.0 (fully active) to specific viseme targets based on the current speech timestamp.
  • Coarticulation and Blending: Natural human speech blends sounds together. Instead of switching abruptly between shapes, the engine calculates overlapping curves using techniques like linear interpolation (lerp), spherical interpolation, or cubic hermite splines. This ensures adjacent mouth shapes blend smoothly and account for speech velocity.

4. Updating the glTF Runtime Mesh

During playback, the application synchronizes the system clock with the audio playback position. On every animation frame, the runtime (such as Three.js, Babylon.js, or custom WebGL/WebGPU engines) updates the character:

  1. Query Timestamps: The engine queries the current audio playback position.
  2. Evaluate Weights: Interpolated weight values for all active visemes are calculated for that timestamp.
  3. Apply to Node: The values are written directly to the glTF mesh node's weights array or the rendering engine's corresponding morphTargetInfluences property.
  4. Vertex Shader Execution: The GPU computes the final vertex positions on the fly by adding the base position attribute to the product of each target’s displacement vector and its corresponding scalar weight:

\[\text{Final Position} = \text{Base Position} + \sum_{i} (\text{Weight}_i \times \text{Displacement}_i)\]

By offloading the geometric deformation to the GPU via morph target weights, the speech synthesis pipeline maintains low CPU overhead and low latency, enabling seamless, frame-accurate lip sync in web and real-time interactive applications.