JPEG Component Interleaving Across Scans Explained

This article examines how the JPEG standard (ITU-T T.81 / ISO/IEC 10918-1) organizes and multiplexes color data by defining component interleaving within and across scans. It covers the structural definition of the Minimum Coded Unit (MCU), the role of the Start of Scan (SOS) marker, how sampling factors govern component layout, and the architectural differences between sequential and progressive multi-component scans.

Sampling Factors and the Minimum Coded Unit (MCU)

In the JPEG specification, an image can consist of up to 255 color components, though common formats utilize three (such as \(Y, C_b, C_r\)). Each component \(i\) is assigned a horizontal sampling factor (\(H_i\)) and a vertical sampling factor (\(V_i\)), ranging from 1 to 4. These factors dictate the relative spatial resolution of each channel.

When components are interleaved, the data is grouped into Minimum Coded Units (MCUs). The dimensions of an MCU are determined by the maximum horizontal and vertical sampling factors across all components present in the scan:

\[H_{max} = \max(H_i), \quad V_{max} = \max(V_i)\]

An MCU represents a spatial region of \((H_{max} \times 8)\) by \((V_{max} \times 8)\) pixels. For each component \(i\) included in an interleaved scan, that component contributes \(H_i \times V_i\) individual \(8 \times 8\) data units (blocks) to the MCU.

Interleaved vs. Non-Interleaved Scans

The Start of Scan (SOS) marker segment controls how components are distributed throughout the bitstream by specifying \(N_s\), the number of image components in the current scan (ranging from 1 to 4):

• Non-Interleaved Scans (\(N_s = 1\)): When a scan contains only one component, no interleaving occurs. The MCU degenerates into a single \(8 \times 8\) data unit of that specific component. The entire image plane for that component is encoded sequentially from left to right, top to bottom, before another scan begins.
• Interleaved Scans (\(N_s > 1\)): When two or more components are grouped into the same scan, they are interleaved at the MCU level. Instead of completing an entire component plane before moving to the next, the encoder alternates between components within every MCU across the scan.

Within each interleaved MCU, the data units are serialized in order of their component specification in the SOS header:

1. All \(H_1 \times V_1\) blocks of Component 1, traversed in raster order.
2. All \(H_2 \times V_2\) blocks of Component 2, traversed in raster order.
3. This sequence continues for up to 4 components per scan.

Example: Standard 4:2:0 Interleaving

In a typical \(Y C_b C_r\) image with 4:2:0 chroma subsampling:

• Luminance (\(Y\)): \(H_1 = 2, V_1 = 2\)
• Chrominance (\(C_b\)): \(H_2 = 1, V_2 = 1\)
• Chrominance (\(C_r\)): \(H_3 = 1, V_3 = 1\)

Here, \(H_{max} = 2\) and \(V_{max} = 2\). Each MCU covers a \(16 \times 16\) pixel area. In an interleaved scan containing all three components (\(N_s = 3\)), a single MCU consists of 6 total blocks ordered as: \[[Y_{0,0}, Y_{0,1}, Y_{1,0}, Y_{1,1}, Cb_{0,0}, Cr_{0,0}]\]

The decoder reconstructs these six \(8 \times 8\) frequency-domain blocks, applies the inverse discrete cosine transform (IDCT), and maps them to the corresponding \(16 \times 16\) spatial region before advancing to the next MCU.

Interleaving Rules in Progressive Scans

The JPEG standard applies strict structural constraints on component interleaving when using progressive DCT-based processes:

1. DC Coefficient Scans: Multiple components can be interleaved within a single scan only if the spectral selection is restricted exclusively to DC coefficients (spectral selection start \(Ss = 0\) and end \(Se = 0\)). This permits decoders to quickly establish a coarse, full-color thumbnail of the image using an interleaved stream of DC values.
2. AC Coefficient Scans: Interleaving across multiple components is strictly prohibited for scans containing AC coefficients (\(Ss > 0\) or \(Se > 0\)). Every AC spectral selection or successive approximation scan must have \(N_s = 1\). Consequently, progressive JPEG files interleave components only during the initial DC pass; all subsequent detail passes are strictly non-interleaved.