Three.js VR Hand Tracking with XRHandModelFactory

This article provides a step-by-step guide on how to implement fully articulated, realistic hand tracking in a Three.js WebXR scene using the XRHandModelFactory. You will learn how to configure the WebXR renderer, load the necessary hand tracking modules, instantiate the factory, and map physical hand movements to 3D mesh models in real time.

Prerequisites and Imports

To implement hand tracking, you need a WebXR-compatible VR headset that supports hand tracking (such as the Meta Quest series) and a basic Three.js scene. You must import the core Three.js library, the VRButton to enable VR mode, and the XRHandModelFactory to generate the 3D hand representations.

import * as THREE from 'three';
import { VRButton } from 'three/addons/webxr/VRButton.js';
import { XRHandModelFactory } from 'three/addons/webxr/XRHandModelFactory.js';

Step 1: Initialize the Renderer and Enable WebXR

Your WebGLRenderer must have XR support enabled. This allows Three.js to interface with the device’s WebXR API.

const renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.setSize(window.innerWidth, window.innerHeight);
renderer.xr.enabled = true;
document.body.appendChild(renderer.domElement);

// Add the VR Button to the DOM
document.body.appendChild(VRButton.createButton(renderer));

Step 2: Set Up the Hand Model Factory

The XRHandModelFactory manages the creation of the visual hand models. It automatically detects the joints provided by the WebXR API and maps them to a 3D mesh. You can instantiate it once and use it for both hands.

const handModelFactory = new XRHandModelFactory();

Step 3: Configure the Left and Right Hands

WebXR distinguishes between controllers and hands. To capture joint data, you must retrieve the hand input sources using renderer.xr.getHand(index). The index 0 typically represents the left hand, and 1 represents the right hand.

For each hand, you retrieve the hand object, generate a visual model using the factory, add the model to the hand object, and finally add the hand object to your scene.

// Left Hand
const hand1 = renderer.xr.getHand(0);
scene.add(hand1);

const handModel1 = handModelFactory.createHandModel(hand1, 'mesh');
hand1.add(handModel1);

// Right Hand
const hand2 = renderer.xr.getHand(1);
scene.add(hand2);

const handModel2 = handModelFactory.createHandModel(hand2, 'mesh');
hand2.add(handModel2);

The second argument in createHandModel defines the profile style. The default 'mesh' profile renders a realistic, fully deformed skin mesh. You can also use 'spheres' or 'boxes' for a joint-by-joint visualization which is useful for debugging.

Step 4: Add Lights and Render

Because the default hand models use shaded materials, your scene must contain light sources for the hands to be visible.

const ambientLight = new THREE.AmbientLight(0xffffff, 0.5);
scene.add(ambientLight);

const directionalLight = new THREE.DirectionalLight(0xffffff, 0.8);
directionalLight.position.set(0, 10, 0);
scene.add(directionalLight);

// Standard Three.js animation loop modified for WebXR
renderer.setAnimationLoop(() => {
    renderer.render(scene, camera);
});

Once the VR session starts and the user looks at their hands, the WebXR API will begin sending joint data, and XRHandModelFactory will automatically animate the 3D hands to match the user’s real-world movements.