Access WebXR Depth Sensing Data in Three.js

This article explains how to retrieve and utilize real-world depth sensing data within an augmented reality (AR) session using Three.js and the WebXR Device API. You will learn how to configure your WebXR session to request depth data, access depth buffers on both the CPU and GPU, and use this information to create realistic environment occlusions and physics interactions in your web-based AR applications.

1. Enable Depth Sensing in the WebXR Session

To access depth data, you must request the depth-sensing feature when initializing your WebXR session. In Three.js, this is configured using the options passed to the ARButton.

You must specify your preferences for both usage (CPU or GPU optimized) and data format (such as luminance-alpha or float32).

import { ARButton } from 'three/addons/webxr/ARButton.js';

const sessionInit = {
  requiredFeatures: ['depth-sensing'],
  depthSensing: {
    usagePreference: ['gpu-optimized', 'cpu-optimized'],
    formatPreference: ['luminance-alpha', 'float32']
  }
};

document.body.appendChild(ARButton.createButton(renderer, sessionInit));

2. Access the Depth Information in the Render Loop

Once the session is active, you can access depth data on every frame via the WebXR frame object. You must retrieve the active viewer pose and request the depth information for each specific view (eye).

function onWindowLoop(timestamp, frame) {
  if (frame) {
    const session = renderer.xr.getSession();
    const referenceSpace = renderer.xr.getReferenceSpace();
    const pose = frame.getViewerPose(referenceSpace);

    if (pose) {
      for (const view of pose.views) {
        // Access depth information for the current view
        const depthInfo = frame.getDepthInformation(view);
        if (depthInfo) {
          processDepthData(depthInfo);
        }
      }
    }
  }
  renderer.render(scene, camera);
}

renderer.setAnimationLoop(onWindowLoop);

3. Handling CPU vs. GPU Depth Data

Depending on the hardware capability and the preferences you defined, the browser will return either CPU-based depth data (XRCPUDepthInformation) or GPU-based depth data (XRGPUTextureDepthInformation).

CPU Depth Data (For Physics and Raycasting)

If the session returns CPU depth data, you can directly query depth values in meters at specific coordinates. This is ideal for collision detection and surface snapping.

if (depthInfo instanceof XRCPUDepthInformation) {
  // Get depth in meters at the center of the viewport (x: 0.5, y: 0.5)
  const depthInMeters = depthInfo.getDepthInMeters(0.5, 0.5);
  console.log(`Depth at center: ${depthInMeters} meters`);
}

GPU Depth Data (For Shaders and Occlusion)

If the session returns GPU depth data, it provides a WebGL texture containing the depth map. This is optimized for real-time rendering effects, such as hiding virtual objects behind physical obstacles (occlusion).

if (depthInfo instanceof XRGPUTextureDepthInformation) {
  const depthTexture = depthInfo.texture;
  
  // To use this in Three.js, wrap the raw WebGLTexture
  const properties = renderer.properties.get(threeDepthTexture);
  properties.__webglTexture = depthTexture;
  
  // Assign the texture to a custom RawShaderMaterial for rendering
  customMaterial.uniforms.uDepthTexture.value = threeDepthTexture;
}

4. Normalizing Depth Coordinates

WebXR depth textures may not align perfectly with your viewport’s aspect ratio. To map the depth texture to your screen space inside custom shaders, you must use the coordinate transform matrix provided by the depthInfo object.

// Get the conversion matrix
const normDepthCoordFromNormViewCoords = depthInfo.normDepthBufferFromNormView.matrix;

// Pass this matrix as a uniform to your vertex or fragment shader
customMaterial.uniforms.uDepthMatrix.value = normDepthCoordFromNormViewCoords;