What Causes Quantization Error in Digital Signals

Quantization error is the inherent discrepancy that occurs during analog-to-digital conversion when an infinitely variable, continuous physical signal is rounded to the nearest discrete value supported by a digital system. Because binary systems rely on a finite number of bits to represent data, they cannot capture the infinite precision of real-world phenomena like sound, temperature, or voltage. This article explains the mechanics of the quantization process, the mathematical origin of this error, and how digital resolution limits signal fidelity.

The Continuous vs. Discrete Dilemma

Physical signals exist in a continuous state, meaning they can assume an infinite number of values within any given range. For example, a voltage signal moving between 0V and 5V does not simply jump between whole numbers; it passes through infinite fractional values such as 2.3479…V.

Digital processors, however, operate in a binary format using bits (\(0\) and \(1\)). A binary system with \(N\) bits can only represent a finite number of unique states, calculated as \(2^N\). For instance, an 8-bit system can represent \(2^8 = 256\) distinct levels, while a 16-bit system can represent \(2^{16} = 65,536\) levels. Because infinity cannot be mapped into a finite set without data loss, the continuous signal must be approximated.

The Quantization Process and Step Size

When an Analog-to-Digital Converter (ADC) samples a signal, it divides the total input range into a series of uniform discrete bins or step sizes, often denoted as \(\Delta\) or \(Q\). The step size is determined by the formula:

\[Q = \frac{V_{\text{max}} - V_{\text{min}}}{2^N}\]

Where \(V_{\text{max}} - V_{\text{min}}\) represents the full-scale amplitude range, and \(N\) is the bit resolution.

During the conversion, the ADC measures the instantaneous continuous voltage and assigns it to the closest discrete quantization level. Any value falling within a given step interval is forced to round up or down to that interval’s assigned binary code.

How the Error Arises

Quantization error (\(e_q\)) is the direct difference between the actual continuous input voltage (\(x\)) and the resulting quantized output value (\(x_q\)):

\[e_q = x - x_q\]

Because the digital system rounds the input to the nearest available step, the maximum error in a standard rounding system is half of the step size:

\[-\frac{Q}{2} \le e_q \le \frac{Q}{2}\]

This rounding process discards the fractional details of the original signal. The lost information manifests as an unwanted noise-like signal, commonly referred to as quantization noise or quantization distortion.

Reducing Quantization Error

While quantization error cannot be completely eliminated in digital systems, it can be minimized using specific engineering techniques:

• Increasing Bit Depth: Adding more bits exponentially increases the number of available discrete levels, making the step size \(Q\) much smaller. For every 1-bit increase in resolution, the signal-to-quantization-noise ratio (SQNR) improves by approximately 6 dB.
• Oversampling: Sampling the signal at frequencies far higher than the Nyquist rate spreads the quantization noise over a wider bandwidth, allowing digital filters to remove the noise outside the target frequency band.
• Dithering: Adding a microscopic amount of randomized noise to the analog signal before conversion breaks up harmonic distortions caused by repetitive rounding errors, converting harsh quantization artifacts into benign, uniform background hiss.