How Cameras Convert Bayer RAW to JPEG YCbCr

Modern digital cameras capture light as single-channel electrical charges filtered through a color pattern, producing raw Bayer data rather than a viewable color picture. To convert this data into a standard JPEG file, the camera's Image Signal Processor (ISP) runs an automated hardware pipeline that demosaics the pixels into full red, green, and blue values, balances and transforms those colors to match human vision, applies non-linear gamma curves, and ultimately converts the RGB signals into luminance and chrominance (YCbCr) components optimized for compression.

1. Pre-Processing the Raw Sensor Data

Before color processing begins, the ISP corrects physical imperfections introduced by the sensor and lens. At this stage, each photosite only records a single intensity value behind a red, green, or blue filter (typically in a 2x2 grid of 50% green, 25% red, and 25% blue). The camera executes:

• Black Level Correction: Subtracts the baseline electrical noise (dark current) inherent to the sensor.
• Defective Pixel Correction: Interpolates values for stuck or dead sensor pixels using neighboring photosite data.
• Lens Shading Correction: Compensates for optical vignetting and color falloff near the edges of the sensor.

2. Demosaicing (Color Filter Array Interpolation)

Because each pixel only captures one color component, the camera must infer the missing two channels for every location. Demosaicing algorithms analyze spatial gradients and edge directions to interpolate the missing colors without creating visual artifacts like moiré or chromatic fringing. Sophisticated edge-aware algorithms, such as Adaptive Homogeneity-Directed (AHD) interpolation, evaluate green channels first (which carry the majority of luminance detail) and then reconstruct the red and blue channels relative to the green channel. The output of this stage is a full, three-channel linear RGB image.

3. White Balance and Color Correction Matrix (CCM)

A camera sensor's color sensitivity does not match the human eye, nor does it inherently adjust to different lighting conditions.

First, the ISP applies digital gains to the red and blue channels relative to the green channel to achieve white balance, ensuring neutral surfaces appear gray or white regardless of the ambient light spectrum (such as tungsten or daylight).

Next, the ISP applies a 3x3 Color Correction Matrix (CCM). This matrix transforms the sensor’s native, camera-specific RGB primaries into a standard color space, such as linear sRGB or Rec.709. The CCM compensates for the spectral cross-talk between the micro-filters on the Bayer array, mapping captured wavelengths into standard target chromaticities.

4. Tone Mapping and Gamma Correction

Raw camera sensor data is strictly linear: double the photons hitting the photodiode equals double the numerical signal. However, human perception of brightness is non-linear and sensitive to changes in shadow tones.

To prepare the image for display and storage:

1. Dynamic Range Compression / Tone Curve: Compresses high-contrast scene data to preserve details in both shadows and highlights.
2. Gamma Correction: Converts linear RGB to non-linear R'G'B' using standard transfer functions (such as the sRGB curve, approximately equivalent to a 2.2 gamma). This maps more digital bit depth into the darker tones where the human eye detects fine gradations.

5. Color Space Conversion to YCbCr

JPEG compression relies on separating brightness information from color information. Human vision has significantly higher spatial acuity for luminance (brightness) than for chrominance (color).

Using the non-linear R'G'B' values, the ISP applies a color difference matrix defined by standard specifications (most commonly ITU-R BT.601 for standard JPEGs):

• Y (Luma): Represents the achromatic brightness detail. \[Y = 0.299R' + 0.587G' + 0.114B'\]
• Cb (Chroma Blue): Represents the difference between blue and luma. \[Cb = -0.1687R' - 0.3313G' + 0.5B' + 128\]
• Cr (Chroma Red): Represents the difference between red and luma. \[Cr = 0.5R' - 0.4187G' - 0.0813B' + 128\]

The offset of 128 centers the chroma values in an 8-bit unsigned integer range (0–255).

6. Chroma Subsampling and Compression

Once transformed into YCbCr, the ISP typically applies chroma subsampling (usually 4:2:0 or 4:2:2). In a 4:2:0 scheme, color resolution is halved both horizontally and vertically, discarding 75% of the chroma data while retaining 100% of the luma (Y) data. Because human perception is insensitive to fine color details, this step reduces data bandwidth with negligible perceptual loss.

Finally, the subsampled YCbCr blocks pass directly into the camera's hardware JPEG encoder, where they undergo Discrete Cosine Transform (DCT), quantization, and entropy (Huffman) encoding to be packaged as the final .jpg file.