YUV to RGB Rounding Errors in Decoded AVIF

AVIF images store color data using YUV (typically YCbCr) representations to maximize compression efficiency, requiring decoders to transform these values back to RGB for display. Because the conversion involves floating-point matrix multiplication implemented via fixed-point or truncated arithmetic, small precision losses inevitably occur. In decoded AVIF images, these rounding errors manifest primarily as subtle color banding in smooth gradients, slight hue drift in neutral tones, and edge fringing, especially in lower bit-depth formats.

The Conversion Mechanics

The AV1 video codec underlying AVIF separates luma (brightness, \(Y\)) from chroma (color difference, \(Cb\) and \(Cr\)). Converting this signal back to standard RGB requires matrix multiplication standardized by color specifications like BT.709, BT.601, or BT.2020. The standard BT.709 transformation formula illustrates the non-integer nature of the coefficients:

• \(R = Y + 1.5748 \times (Cr - \text{offset})\)
• \(G = Y - 0.1873 \times (Cb - \text{offset}) - 0.4681 \times (Cr - \text{offset})\)
• \(B = Y + 1.8556 \times (Cb - \text{offset})\)

Because real-time decoders—particularly browser implementations and GPU hardware pipelines—prioritize speed, they replace floating-point operations with fixed-point integer arithmetic and bit-shifting. Truncation or naive rounding during these matrix operations produces reconstructed RGB values that deviate by \(\pm 1\) or more code values from the original source.

Visual Manifestations in Decoded Images

1. Banding in Smooth Gradients

The most noticeable effect of conversion rounding is color contouring or false banding across flat, gradual transitions such as skies, shadows, or computer-generated vignettes. When the conversion rounds adjacent luma-chroma pairs to identical or discontinuous RGB values, continuous color ramps degrade into distinct, visible bands. This problem is particularly acute in 8-bit AVIF files, where single-value quantization steps represent perceptible jumps to the human eye.

2. Hue Drift in Neutral Tones

True grayscale requires identical RGB values (\(R = G = B\)). In YUV, grayscale occurs when \(Cb\) and \(Cr\) sit precisely at their neutral midpoint (128 in an 8-bit scale). If the conversion matrix does not balance symmetrically due to integer truncation, the calculated \(R\), \(G\), and \(B\) values diverge slightly. Consequently, clean grays or pale skin tones can display a faint green, magenta, or cyan tint in decoded outputs.

3. Out-of-Gamut Clamping and Detail Loss

The conversion formula occasionally yields RGB values below 0 or above the maximum integer threshold (255 for 8-bit, 1023 for 10-bit), especially in highly saturated highlights and deep shadows. When these values are hard-clamped into the valid integer range, subtle texture and highlight details are lost. Clamping artifacts manifest as blotchy, flat patches in saturated areas.

4. Amplification of Chroma Subsampling Artifacts

Most AVIF files use 4:2:0 subsampling, where color resolution is halved horizontally and vertically relative to brightness. Decoders must interpolate chroma before converting to RGB. Rounding errors interact with this interpolation process, accentuating high-contrast boundaries. This appears as colored halos, jagged borders, or chromatic ringing along sharp edges separating contrasting colors.

Prevention and Mitigation

• Adopt 10-bit Encoding: Storing AVIF files in 10-bit precision expands the available code values fourfold (\(0\)–\(1023\)). Rounding errors then occur well below the visual threshold of human perception, virtually eliminating banding and drift upon 8-bit display conversion.
• Decoder Precision: Software decoders that utilize higher-precision intermediate registers (such as 16-bit integers or 32-bit floating-point math) prior to final display quantization avoid compound rounding errors during intermediate transformation steps.
• RGB / Identity Matrix Encoding: For graphics requiring pixel-perfect color fidelity, AVIF supports 4:4:4 sampling using an Identity matrix (matrix_coefficients = 0). This bypasses the YUV-to-RGB conversion step altogether, eliminating conversion errors at the expense of higher file sizes.