JPEG DCT Quantization Formula Explained

Quantization is the primary lossy stage in JPEG image compression, responsible for drastically reducing file size by discarding high-frequency visual details that the human eye cannot easily perceive. After an image block is transformed from the spatial domain to the frequency domain using the Discrete Cosine Transform (DCT), each resulting frequency coefficient is scaled and rounded using a standardized or custom quantization matrix. This guide outlines the exact mathematical formula used to compute quantized JPEG DCT coefficients and explains how each component operates.

The Quantization Formula

The mathematical formula for quantizing an 8x8 block of DCT coefficients is:

\[F_Q(u, v) = \text{round}\left( \frac{F(u, v)}{Q(u, v)} \right)\]

Where:

• \(u, v\) are the horizontal and vertical spatial frequency coordinates within the 8x8 block, ranging from \(0\) to \(7\).
• \(F(u, v)\) is the unquantized DCT coefficient at coordinate \((u, v)\).
• \(Q(u, v)\) is the value at coordinate \((u, v)\) from the predefined Quantization Table (matrix).
• \(\text{round}()\) represents rounding to the nearest integer, mathematically defined for positive values as \(\lfloor x + 0.5 \rfloor\) (or \(\text{sign}(x) \cdot \lfloor |x| + 0.5 \rfloor\) for signed values).
• \(F_Q(u, v)\) is the resulting quantized DCT coefficient.

How the Components Function

1. DCT Coefficients (\(F\)): The coefficient at \((0, 0)\) is the DC coefficient, representing the average brightness of the 64-pixel block. The remaining 63 values are AC coefficients, which represent increasingly higher horizontal, vertical, and diagonal frequencies.

2. Quantization Matrix (\(Q\)): The quantization table contains predetermined step sizes. Lower frequencies typically have smaller values in \(Q(u, v)\) to preserve detail, while higher frequencies have much larger values. Because human vision is less sensitive to high-frequency chrominance and luminance variations, dividing by larger divisors reduces these higher-frequency coefficients to zero or near-zero.

3. The Rounding Operation: The division \(\frac{F(u, v)}{Q(u, v)}\) yields floating-point numbers. Rounding forces these values into discrete integers. This rounding step is non-invertible and is the exact point where data loss occurs in the JPEG compression pipeline.

Reconstruction (Dequantization)

During image decompression, the decoder reconstructs the approximate DCT coefficients using the inverse operation:

\[F'(u, v) = F_Q(u, v) \times Q(u, v)\]

Because the fractional component was discarded during the rounding phase, \(F'(u, v)\) approximates \(F(u, v)\) but rarely matches it identically, resulting in the compression artifacts characteristic of JPEG images at lower quality settings.