Render VR Controllers in Three.js with XRControllerModelFactory

This article provides a step-by-step guide on how to render realistic, hardware-specific VR hand controllers in a Three.js WebXR scene using the XRControllerModelFactory. You will learn how to import the factory, retrieve controller grip spaces from the renderer, generate dynamic 3D models based on the user’s active VR hardware, and add them to your virtual environment.

Step 1: Import the Required Modules

To get started, you must import the core Three.js library along with the XRControllerModelFactory. This utility is located in the examples folder of the Three.js package.

import * as THREE from 'three';
import { XRControllerModelFactory } from 'three/examples/jsm/webxr/XRControllerModelFactory.js';

Step 2: Enable WebXR on the Renderer

Before handling controllers, ensure your WebGLRenderer has WebXR enabled. Without this, Three.js will not process VR inputs or camera tracking.

const renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.setSize(window.innerWidth, window.innerHeight);
renderer.xr.enabled = true; // Enables VR rendering capabilities
document.body.appendChild(renderer.domElement);

Step 3: Understand Controller Grip vs. Controller Pointer

Three.js provides two distinct WebXR controller objects for each hand: * Controller Pointer (getController): Used for pointing, raycasting, and selecting UI elements (renders a line or pointer). * Controller Grip (getControllerGrip): Represents the physical position and orientation of the user’s hand holding the controller. This is where you attach the 3D controller models.

Step 4: Instantiate the Factory and Create Controller Models

Initialize the XRControllerModelFactory. This factory automatically queries the WebXR device API to fetch the correct 3D asset matching the user’s connected VR headset (e.g., Meta Quest, HTC Vive, or Valve Index).

// Initialize the factory
const controllerModelFactory = new XRControllerModelFactory();

// Set up the first controller (index 0)
const controllerGrip1 = renderer.xr.getControllerGrip(0);
const model1 = controllerModelFactory.createControllerModel(controllerGrip1);
controllerGrip1.add(model1);
scene.add(controllerGrip1);

// Set up the second controller (index 1)
const controllerGrip2 = renderer.xr.getControllerGrip(1);
const model2 = controllerModelFactory.createControllerModel(controllerGrip2);
controllerGrip2.add(model2);
scene.add(controllerGrip2);

Step 5: Handle Controller Connection and Disconnection (Optional)

The XRControllerModelFactory automatically manages loading and unloading models when controllers connect or disconnect. However, you can listen for these events directly on the controller objects if you need to trigger custom visual feedback in your scene.

controllerGrip1.addEventListener('connected', (event) => {
    console.log('Controller connected: ', event.data);
});

controllerGrip1.addEventListener('disconnected', () => {
    console.log('Controller disconnected');
});