How Mobile AR Engines Calculate glTF Reflections

This article examines how mobile Augmented Reality (AR) engines, such as Apple ARKit and Google ARCore, analyze real-world surroundings to project dynamic, realistic environmental reflections onto glTF 3D assets. It covers the end-to-end pipeline: capturing camera data, generating dynamic environment probes, extracting spherical harmonics and specular maps, and rendering these data streams using Physically Based Rendering (PBR) shaders compatible with the glTF 2.0 standard.

1. Real-World Environmental Capture

The process begins with the mobile device's camera stream. As the user moves the device, AR engines continuously map spatial features and collect color, exposure, and intensity data from the physical environment.

Because a standard camera feed only captures a narrow field of view, AR platforms use machine learning models and spatial tracking to infer the unobserved environment:

  • ARKit (Environment Probes): Places reflection probes at specific 3D anchors in the scene. As the camera moves, it stitches video frames onto a virtual bounding sphere or cube. Deep learning models fill in the unseen areas (such as the ceiling or behind the user) to create a continuous 360-degree environment.
  • ARCore (Environmental HDR): Uses machine learning to predict lighting in all directions from a single camera frame, outputting real-time spherical harmonics and an estimated cubemap representing the surrounding physical space.

2. Converting Sensor Data to Image-Based Lighting (IBL)

To project light and reflections onto virtual assets, the 360-degree environmental data is split into two components: diffuse irradiance and specular radiance.

Diffuse Irradiance (Spherical Harmonics)

Diffuse light represents ambient, non-directional illumination bouncing off matte surfaces. AR engines approximate this using Spherical Harmonics (SH)—typically L2 (9 coefficients) or L3 (16 coefficients). Instead of passing a large texture to the graphics pipeline, the engine supplies a compact mathematical representation of ambient color and brightness, drastically reducing mobile GPU overhead.

Specular Radiance (Prefiltered Environment Maps)

Specular reflections represent the sharp, mirror-like reflections on shiny surfaces. Mobile engines project the stitched 360-degree imagery into a dynamic cubemap. To allow for varying surface smoothness, the cubemap is run through a prefiltering algorithm that generates mipmaps. Higher mip levels store increasingly blurred versions of the environment, corresponding to surface roughness.

3. Mapping Lighting Data to the glTF PBR Pipeline

The glTF 2.0 format relies on a metallic-roughness PBR workflow. When a glTF asset is rendered in the AR scene, its fragment shader computes surface reflections by querying the environment data:

  1. Fresnel Calculation: The shader calculates how reflective the surface appears based on the viewing angle (the Schlick approximation of the Fresnel effect). Grazing angles produce stronger reflections than perpendicular views.
  2. Roughness Sampling: The shader reads the roughnessFactor and the roughness texture channel of the glTF material. This determines which mip level of the prefiltered environment cubemap to sample. A roughness value near 0.0 samples the crisp base mip level (mirror-like), while higher roughness values sample blurrier mip levels.
  3. Metallic Blending: The metallicFactor controls how the base color interacts with reflections. Non-metals (dielectrics) reflect the environment's true colors while preserving their own diffuse albedo, while metals tint the reflection with the material's base color and eliminate diffuse shading.

4. Real-Time Performance Optimizations

Running real-time reflections on mobile hardware requires strict performance balancing:

  • Asynchronous Probe Generation: Updating a full 360-degree cubemap every frame causes thermal throttling and frame drops. AR engines often update probe textures at lower refresh rates (e.g., 10 to 15 Hz) or across alternating frames, while the visual renderer continues at 60 or 120 FPS.
  • Temporal Filtering: To prevent sudden popping when the camera detects new lighting conditions, engines interpolate between old and new environment probes smoothly.
  • Low-Resolution Textures: Specular cubemaps in mobile AR rarely exceed 256x256 or 512x512 pixels per face. The inherent blur on rough materials conceals the lower resolution, while high-frequency details on smooth surfaces are supplemented by screen-space reflections or static high-detail fallbacks when appropriate.