Eye-Tracking Calibration for Spatial Computing Games

This article outlines how spatial computing developers implement eye-tracking calibration pipelines to power responsive gaze-directed user interfaces. It covers obtaining raw sensor data through modern XR APIs, executing multi-point calibration routines to correct for anatomical differences, applying real-time data smoothing to eliminate jitter, and structuring spatial UI colliders to deliver precise, low-latency gaze selection.

1. Accessing Hardware via XR APIs

Implementing eye-tracking starts with interfacing with the headset’s eye-tracking hardware through platform-standard runtimes. Most spatial computing platforms rely on OpenXR standards, specifically the XR_EXT_eye_gaze_interaction extension. In engines like Unity or Unreal Engine, developers use platform-specific SDKs (such as visionOS, Meta XR Core SDK, or Vive OpenXR) to query the user's eye pose.

The system returns a combined gaze pose or individual eye poses consisting of:

  • Origin: A 3D spatial coordinate indicating the center point between the eyes or the pupil centers.
  • Direction: A normalized 3D forward vector showing where the eyes are looking relative to the headset coordinate space.

2. The Multi-Point Calibration Process

Anatomical differences—such as interpupillary distance (IPD) and the angle between the optical axis (pupil center) and the visual axis (fovea alignment, known as the kappa angle)—cause raw gaze data to drift. Developers implement calibration routines to correct this variance:

  1. Point Presentation: The user is prompted to look at a series of calibration targets (typically 5 or 9 points) distributed across the field of view in a 3D grid.
  2. Fixation Verification: The system verifies stable fixation by checking that the user's head position remains relatively still and that gaze velocity drops below a set threshold for a designated duration (e.g., 200–400 ms).
  3. Data Collection: Gaze samples are recorded and matched against the known spatial coordinates of the target markers.
  4. Polynomial Regression / Mapping Matrix: Developers apply geometric or polynomial transformation algorithms to map the raw sensor vectors to true world coordinates, generating an offset matrix that is stored and applied to all subsequent real-time gaze vectors.

3. Noise Filtering and Gaze Smoothing

Human eyes constantly produce micro-saccades, nystagmus, and rapid drift, which makes raw gaze vectors unsuitable for driving steady UI focus. To make the interface feel responsive rather than jittery, developers run raw data through real-time filtering algorithms:

  • The 1€ (One Euro) Filter: A preferred choice for spatial interactions because it dynamically adjusts smoothing based on movement speed. It heavily smooths low-velocity fixations to keep UI highlights stable while minimizing lag during high-velocity saccades when the user switches targets.
  • Kalman Filtering: Predicts the next gaze point based on past trajectory, which reduces perceived latency at the cost of slight mathematical overhead.
  • Dwell Windows: Brief temporal buffers (e.g., 50–100 ms) prevent UI elements from flickering when the user's gaze rapidly brushes across intermediate buttons.

4. Gaze-Target Raycasting and Collider Architecture

Once the calibrated and filtered gaze vector is generated, it must be matched to interface elements in the 3D scene. Traditional single-pixel raycasting makes UI selection frustrating because spatial UI requires excessive precision. Developers solve this using tailored collision techniques:

  • Cone Casting and Sphere Casting: Instead of casting an infinitesimally thin ray, developers cast a cone (typically 1.5° to 3° wide, matching the human foveal field) or a swept sphere. The system selects the UI element closest to the center of the cone.
  • Expanded Hit Targets: Interactive UI components are built with invisible colliders that extend 20% to 30% beyond their visible visual boundaries, allowing users to trigger hover states without aiming precisely at visual edges.
  • Magnetic Target Snapping: When a gaze ray passes within proximity of an interactive element, the active focus point softly snaps to that element's center, preventing UI focus loss caused by natural eye tremors.

5. Solving the Midas Touch with Multi-Modal Input

To prevent inadvertent clicks caused by simply looking at an element (the "Midas Touch" problem), gaze is rarely used as a standalone activation trigger. Instead, the calibrated gaze acts as the pointing mechanism, while selection is confirmed through a secondary action:

  • Indirect Pinch: The gaze highlights the UI element, and an micro-gesture (like touching the index finger to the thumb) triggers the event.
  • Dwell Triggers: For accessibility or controller-free contexts, keeping the gaze inside the target collider for a set duration (typically 400–800 ms) fills a radial progress bar to trigger the action.
  • Hardware Confirmation: Gamepad triggers, wrist gestures, or voice commands confirm the action on the currently focused element.