How AR Games Handle Dynamic Real-World Occlusions

In augmented reality (AR) game development, maintaining immersion requires virtual objects to be realistically hidden when physical obstacles—particularly moving people—pass in front of them. This article outlines the primary engineering pipelines developers use to resolve dynamic real-world occlusions. By combining real-time hardware depth sensing, machine learning-driven segmentation, and standard graphics depth-buffer manipulation, AR engines can accurately mask virtual assets behind dynamic physical entities in real time.

Real-Time Depth Sensing Hardware

Modern mobile and headset hardware captures physical spatial data using specialized depth sensors, such as Time-of-Flight (ToF) cameras and LiDAR scanners. These sensors emit light signals to measure the distance between the device and physical surfaces for every frame. The raw output is converted into a real-time depth map, which provides a per-pixel distance value of the physical environment, including moving humans.

AI and Semantic Person Segmentation

Because hardware depth sensors often operate at lower resolutions or struggle with fine edges like hair and fingers, developers utilize on-device neural networks for semantic human segmentation. Frameworks such as Apple ARKit (People Occlusion) and Google ARCore (Depth API) use computer vision models trained specifically to identify human bodies in the RGB camera feed.

The model generates two distinct outputs:

  • Human Stencil: A binary or grayscale 2D mask that isolates human silhouettes from the background.
  • Human Depth Map: An estimated per-pixel distance map specifically calculated for the detected human body.

Depth Buffer (Z-Buffer) Integration

To blend physical and virtual elements seamlessly, game engines like Unity and Unreal Engine inject physical depth data directly into the rendering pipeline:

  1. Occluder Geometry: The engine projects a screen-space mesh or screen-aligned quad that represents the camera feed.
  2. Writing to the Z-Buffer: The engine writes the physical human depth values into the graphics pipeline's depth buffer (Z-buffer) before rendering the virtual game objects.
  3. Depth Testing: When virtual entities (such as characters, props, or UI elements) are rendered, standard depth testing evaluates whether the virtual object's Z-value is greater (further away) than the value in the depth buffer. If the physical person is closer to the camera than the virtual entity, the virtual pixels fail the depth test and are discarded.

Edge Blending and Temporal Smoothing

Raw segmentation boundaries often produce jagged edges or flickering artifacts around moving people. Developers mitigate this using:

  • Alpha Feathering: Softening the edges of the human silhouette mask via screen-space blur or custom shaders to blend physical edges naturally with virtual content.
  • Temporal Filtering: Blending depth values across consecutive frames to eliminate high-frequency jitter caused by fast movements or sensor noise.

Performance Balancing

Dynamic occlusion is computationally intensive. To preserve target frame rates (typically 60 to 90 FPS), developers frequently run segmentation algorithms at a lower resolution (e.g., half-resolution) than the display output, upsampling the result using bilateral filters that preserve sharp depth boundaries without straining the mobile GPU.