How AI Portrait Relighting Uses Normals and Albedo Maps

Artificial intelligence portrait relighting models alter the studio lighting of a photograph after it has been taken by deconstructing the 2D image into fundamental 3D physical properties. This article provides an overview of how deep learning networks estimate surface normals to understand facial geometry and extract albedo maps to isolate true skin and clothing colors. By separating these elements from the original ambient lighting, the AI can digitally introduce new light sources, shadows, and reflections with photorealistic accuracy.

The Core Concept: Inverse Rendering

To change the lighting in a flat photograph, an AI model must perform a process called inverse rendering. When a camera takes a photo, it captures the final result of light bouncing off a 3D subject. Inverse rendering works backward. The AI analyzes the 2D pixels and divides the image into separate layers: geometry (surface normals), base color (albedo), and the original lighting environment.

Estimating Surface Normals for 3D Geometry

A surface normal is an invisible line pointing directly perpendicular to a given point on a surface. In computer graphics, a surface normal map uses red, green, and blue (RGB) colors to tell the computer which direction every single pixel on a subject's face is pointing—whether it is the tip of the nose pointing forward, or the curve of the cheek pointing to the side.

AI models estimate surface normals from a single 2D photo using Convolutional Neural Networks (CNNs) trained on massive datasets of 3D-scanned human faces. Because human faces share a universal structure, the AI learns to recognize visual cues like shading gradients and facial landmarks. When you feed a standard portrait into the model, it predicts the 3D contours of the face and generates a surface normal map. This map tells the relighting engine exactly how new virtual light should wrap around the subject's features and where shadows should fall.

Extracting the Albedo Map for True Color

The albedo map represents the pure, intrinsic color and texture of the subject, completely devoid of any shadows, highlights, or directional lighting. For example, a shadow on a person's jawline makes the skin pixels look dark brown or grey in a standard photo, even though the skin tone is actually uniform.

To estimate the albedo map, the AI must identify and strip away the existing lighting. The neural network compares the original photo against its newly estimated surface normal map. By analyzing how the brightness changes across the predicted 3D curves of the face, the AI calculates where the original light source was located. It then subtracts these highlights and shadows from the image. What remains is a flat, evenly lit texture map containing only the true colors of the skin, eyes, hair, and clothing.

Training the AI with Light Stages

AI models do not guess these maps blindly; they learn to generate them using data captured from a device called a Light Stage. A Light Stage is a spherical dome lined with hundreds of synchronized LED lights and multiple cameras.

A human subject sits inside the dome and is photographed under rapidly changing lighting angles. This process captures the exact ground-truth surface normals and true albedo of the person. Researchers feed millions of these Light Stage image pairs into the AI during training. The AI practices predicting the normal and albedo maps from a single image, constantly checking its work against the real Light Stage data until its estimations are accurate.

Applying the New Studio Lighting

Once the AI has successfully separated the portrait into a surface normal map and an albedo map, the actual relighting takes place using traditional physics-based rendering equations:

  1. Defining the New Light: The user selects a new lighting direction, color, and intensity.
  2. Calculating Shading: The AI projects the new virtual light onto the surface normal map. Pixels facing the new light source are illuminated, while pixels facing away are cast into shadow.
  3. Blending Color: This new shading calculation is multiplied over the albedo map. This restores the person's skin texture, freckles, and hair color under the influence of the new lighting.
  4. Adding Specular Highlights: Advanced AI models also estimate a roughness or specular map to add realistic shine to oily skin, lips, and eyes based on the new light's angle.

By recombining these layers, the AI outputs a final 2D portrait that looks as though the photographer physically moved the studio lights before pressing the shutter.