JPEG Pleno Modalities: Light Fields and Holography

JPEG Pleno is an open standard developed by the Joint Photographic Experts Group to represent, compress, and exchange next-generation, high-dimensional visual content. Beyond conventional two-dimensional images, JPEG Pleno establishes a unified framework that targets several advanced imaging modalities: light fields, point clouds, and digital holography. This article breaks down these targeted modalities, their technological characteristics, and how the standard facilitates seamless processing and rendering for immersive applications.

Light Fields

Light field imaging captures not only the intensity and color of light in a scene, but also the specific direction from which light rays travel through space. By recording this four-dimensional visual data (two spatial dimensions and two angular dimensions), light fields allow viewers to interact with the captured scene dynamically.

JPEG Pleno categorizes light fields into two main formats:

• Lenslet (Dense) Light Fields: Captured using plenoptic cameras equipped with a micro-lens array placed over a conventional image sensor. This setup captures dozens of micro-perspectives in a single exposure, allowing for post-capture digital refocusing and depth extraction.
• Camera Array (High-Density or Sparse) Light Fields: Captured using arrays of synchronized, spatially distributed cameras. These systems provide wider baselines and expanded fields of view, enabling smooth horizontal and vertical motion parallax when rendering views from novel perspectives.

The JPEG Pleno standard defines specialized coding tools capable of handling the high inter-view redundancy inherent to both lenslet and camera-array setups, maximizing compression efficiency while preserving multi-view fidelity.

Point Clouds

While closely aligned with standalone standards like JPEG Pleno Point Cloud and MPEG V-PCC/G-PCC, volumetric point cloud representations are integral to the broader JPEG Pleno ecosystem. A point cloud represents 3D physical surfaces using a dense collection of unstructured points located in three-dimensional coordinates \((X, Y, Z)\), often accompanied by attributes such as color (RGB), surface normals, reflectance, and transparency.

JPEG Pleno approaches point clouds as geometric components capable of interfacing with image-based representations. This modality is critical for augmented reality, autonomous navigation, cultural heritage archiving, and volumetric video, where real-time surface rendering and spatial interaction are required.

Digital Holography

Digital holography represents the most advanced volumetric imaging modality targeted by JPEG Pleno. Unlike light fields and traditional photography, which record only the amplitude (intensity) of light, holography records both the amplitude and the phase of optical wavefronts via interference patterns created with coherent light sources (lasers).

JPEG Pleno standardizes the representation and compression of these holographic wavefronts:

• Continuous Depth Cues: Because phase data is preserved, reconstructed holographic wavefronts provide all visual depth cues naturally, eliminating the vergence-accommodation conflict common in conventional stereoscopic displays.
• Microscopic and Macroscopic Data: JPEG Pleno addresses both digital holographic microscopy (used in biomedical and material science for non-invasive 3D measurements) and macroscopic display holography (targeted at dynamic spatial light modulators and true 3D holographic screens).
• Speckle and Interference Handling: Holographic data exhibits complex, highly sensitive interference patterns that degrade rapidly under standard lossy transform codecs. JPEG Pleno develops phase-aware transformation and quantization methods to compress interference fields without destroying phase integrity.

The Unified System Architecture

A core objective of JPEG Pleno is not merely treating these modalities as isolated formats, but unifying them under a shared system architecture. By establishing standardized metadata layers, color spaces, and coordinate reference systems, the JPEG Pleno framework allows cross-modal conversion—such as synthesizing a digital hologram from a dense light field or projecting a point cloud into a directional ray space—ensuring broad interoperability across emerging spatial computing platforms.