How JPEG XT Stores HDR Floating-Point Data

This article explains the core mechanisms that JPEG XT (ISO/IEC 18477) utilizes to store high dynamic range (HDR) floating-point image data while retaining full backward compatibility with legacy 8-bit JPEG decoders. It covers the two-layer encoding architecture, the role of application marker segments, the use of inverse tone mapping, and the specific residual coding models defined in ISO/IEC 18477-7 for handling IEEE floating-point representations.

Dual-Layer Architecture and Backward Compatibility

JPEG XT handles high dynamic range data through a split dual-layer system composed of a base layer and one or more extension layers.

The Base Layer: Contains a tone-mapped, standard dynamic range (SDR) 8-bit version of the image. It is compressed using standard discrete cosine transform (DCT) baseline JPEG algorithms. Any legacy JPEG decoder can read and render this layer, completely ignoring the HDR data.
The Extension Layer: Contains the auxiliary information and high-precision residual data required to reconstruct the full dynamic range and floating-point precision of the original image.

Storage in APP11 Marker Segments

To ensure that legacy decoders do not fail when reading an HDR file, JPEG XT encapsulates all extension data inside standard JPEG application marker segments—specifically the APP11 marker (payload identifier 0xFFEB).

The APP11 payload includes a four-byte identifier (JPXT), box-structured metadata conforming to ISO/IEC 18477-3, and the bitstream payloads for the extension layers. Legacy decoders encounter the APP11 markers, recognize them as non-critical application metadata, and skip past them to read only the standard frame. Modern JPEG XT decoders parse these segments to extract the tone-mapping parameters and residual streams.

Tone Mapping and Inverse Transform Functions

To encode floating-point values into an 8-bit base layer, a tone mapping operator (TMO) converts high-range radiance values into standard 8-bit non-linear sRGB or YCbCr values. JPEG XT does not mandate a single TMO, but it requires the encoder to store the mathematical inverse of the chosen TMO in the APP11 segment.

JPEG XT Part 7 (ISO/IEC 18477-7), which specifically standardizes floating-point coding, defines several inverse tone mapping functions:

Parametric Curves: Polynomial, rational, or exponential curves with encoded coefficients that model the global luminance mapping.
Look-Up Tables (LUTs): Explicit sample points that map 8-bit base layer code values back to normalized floating-point luminance or color values.

Residual Coding Models for Floating-Point Precision

Once the base layer is mapped back to an intermediate floating-point approximation via the inverse tone mapping function, that approximation is compared to the original HDR source image. The difference is stored as a residual in the extension layer.

JPEG XT Part 7 provides two primary mechanisms for representing and storing floating-point residual data:

1. Multiplicative Residuals (Ratio Modeling)

Because human vision responds logarithmically to light, floating-point luminance spans multiple orders of magnitude. A multiplicative model computes a ratio between the original floating-point sample \(I\) and the reconstructed base-layer sample \(B\):

\[R = \frac{I}{f(B)}\]

Where \(f(B)\) is the intermediate floating-point reconstruction derived from the base layer. The residual \(R\) represents a gain factor. This ratio is typically transformed into a logarithmic representation, quantized, and encoded using DCT blocks within the APP11 stream.

2. Additive Residuals

In certain color spaces or profiles, an additive residual is used:

\[R = I - f(B)\]

Additive residuals are predominantly used when correcting chrominance channels or when encoding fine details where absolute differences are small.

Floating-Point Sample Reconstruction

During decoding, JPEG XT reconstructs floating-point data (typically conforming to the IEEE 754 half-precision 16-bit or single-precision 32-bit formats) through the following pipeline:

Base Layer Decoding: The primary image is decoded using a standard 8-bit IDCT to yield RGB or YCbCr samples.
Intermediate Floating-Point Expansion: The decoded 8-bit samples are passed through the inverse tone mapping function retrieved from the APP11 marker, lifting the integer values into floating-point coordinates.
Extension Layer Decoding: The residual stream from the APP11 segments is entropy-decoded and processed through an inverse DCT.
Synthesis: The decoder applies the residual correction (via multiplication or addition) to the intermediate values.
Color Space Transformation: The final reconstructed values are converted into the target linear floating-point color space (such as scRGB or BT.2020) and stored as IEEE 754 floating-point samples.