Balancing FP8 Formats on Modern AI Hardware

This article provides a comprehensive overview of how 8-bit floating-point (FP8) formats—specifically E4M3 and E5M2—optimize deep learning workloads on modern artificial intelligence accelerators. By reallocating bits between the exponent and the mantissa, these two formats solve the fundamental tension between numerical precision and dynamic range in low-bit computation. Readers will learn the structural differences between E4M3 and E5M2, how modern AI hardware implements them in a hybrid training and inference pipeline, and the necessity of scaling factors to maintain model accuracy while doubling computational throughput.

The Shift to 8-Bit Floating-Point

As deep learning models scale to hundreds of billions of parameters, memory bandwidth and compute efficiency have become primary bottlenecks. Moving from 16-bit formats (like FP16 and BF16) to 8-bit floating-point (FP8) halves memory requirements and doubles the theoretical throughput of matrix multiplication engines on modern hardware, such as NVIDIA Hopper and Blackwell GPUs, AMD Instinct MI300 series, and Intel Gaudi accelerators.

However, squeezing a floating-point number into just eight bits requires strict trade-offs. Every floating-point number consists of three components:

  • Sign bit (1 bit): Determines if the number is positive or negative.
  • Exponent bits (E): Determines the dynamic range (the gap between the smallest and largest representable numbers).
  • Mantissa bits (M): Determines the precision (how finely values can be distinguished within that range).

With one bit reserved for the sign, only seven bits remain. Because a single 8-bit format cannot simultaneously offer high precision and a wide dynamic range, the AI hardware industry standardized on two complementary FP8 formats: E4M3 and E5M2.

E4M3: Prioritizing Precision

The E4M3 format allocates 1 sign bit, 4 exponent bits, and 3 mantissa bits.

By dedicating more bits to the mantissa, E4M3 maximizes numerical precision at the expense of dynamic range. Having 3 mantissa bits allows the format to represent 8 distinct fractional steps between powers of two. To further expand its utility, the standard implementation of E4M3 on AI accelerators deviates slightly from traditional IEEE 754 rules by eliminating the representation for "Infinity." This frees up bit patterns to represent a maximum value of 448.

In neural networks, weights and activations generally follow a predictable, tightly clustered distribution. They rarely experience massive spikes in magnitude, meaning they do not require a massive dynamic range. However, they are highly sensitive to quantization noise. Therefore, E4M3 is the ideal format for storing weights and activations during the forward pass of training and throughout the entire inference process.

E5M2: Prioritizing Dynamic Range

The E5M2 format allocates 1 sign bit, 5 exponent bits, and 2 mantissa bits.

E5M2 sacrifices precision—offering only 4 fractional steps between powers of two—to achieve a much wider dynamic range. Its 5 exponent bits match the exponent allocation of the traditional 16-bit FP16 format, allowing E5M2 to represent values up to 57,344 and down to very small subnormal numbers. Unlike E4M3, E5M2 retains standard IEEE 754 representations for Infinity and NaN (Not a Number).

This wide dynamic range is critical during the backward pass of neural network training. When calculating gradients to update the model, values can vary wildly in magnitude. If a format's range is too narrow, small gradients will underflow to zero (vanishing gradients), and large gradients will overflow to infinity (exploding gradients). E5M2 prevents this data loss, making it the standard choice for gradient calculations.

Hybrid Execution on AI Accelerators

Modern AI hardware does not force developers to choose between E4M3 and E5M2 for an entire workload. Instead, tensor cores and matrix engines are designed to switch between these formats dynamically within the same training loop.

A standard mixed-precision FP8 training step operates as follows:

  1. Forward Pass (E4M3): Model weights and input activations are cast to E4M3. The hardware executes matrix multiplications with high precision, ensuring the network's predictions remain accurate.
  2. Backward Pass (E5M2): Error gradients are computed and cast to E5M2. The hardware processes these gradients using the wider dynamic range, capturing subtle learning signals without underflowing or overflowing.
  3. Master Weights (FP16/BF16/FP32): While the heavy matrix math occurs in FP8, the hardware accumulates the results into higher-precision buffers (usually FP16, BF16, or FP32) to update the master weights, preserving long-term training stability.

Hardware-Accelerated Tensor Scaling

Even with the split between E4M3 and E5M2, an 8-bit range is still narrow compared to 16-bit or 32-bit formats. To fully utilize the available bit representations, modern AI accelerators implement hardware-level tensor scaling.

Before a tensor is quantized into E4M3 or E5M2, the hardware analyzes the maximum absolute value (amax) of the data block. It then multiplies the entire tensor by a scaling factor that shifts the data values into the optimal representable range of the target FP8 format. After the matrix multiplication is complete, the hardware automatically divides the output by the scaling factor to restore the correct magnitude.

By combining this dynamic scaling with the structural strengths of E4M3 and E5M2, modern AI hardware achieves the speed and memory benefits of 8-bit computation while matching the accuracy of 16-bit models.