How Does Catastrophic Cancellation Destroy Digits?

Catastrophic cancellation occurs in classical computing when two nearly equal floating-point numbers are subtracted, causing a severe loss of significant digits and a sharp increase in relative error. Because floating-point numbers store only a fixed number of significant digits, the leading digits of two close values match and cancel each other out, leaving behind only the trailing rounding errors or noise. This phenomenon can turn a seemingly simple subtraction into a major source of numerical inaccuracy in scientific computing, engineering simulations, and financial calculations.

How Floating-Point Representation Works

In classical computers, real numbers are typically represented using the IEEE 754 floating-point standard. A floating-point number is stored in a form similar to scientific notation:

\[x = (-1)^s \times m \times 2^e\]

Here, \(s\) is the sign bit, \(m\) is the significand (or mantissa) containing a fixed number of binary digits, and \(e\) is the exponent. For example, standard 64-bit double-precision floating-point numbers provide 53 bits of significand precision, which corresponds to roughly 15 to 17 decimal digits.

Because the significand has a finite length, most real numbers cannot be represented exactly. Whenever a number is stored or an arithmetic operation is performed, the result is rounded to the nearest representable floating-point value. This introduces a small relative rounding error, bounded by machine epsilon \(\epsilon_{\text{mach}}\).

The Mechanics of Catastrophic Cancellation

Suppose you have two true values, \(a\) and \(b\), that are very close to each other. When they are stored or computed in floating-point arithmetic, they become approximations \(\hat{a}\) and \(\hat{b}\):

\[\hat{a} = a(1 + \delta_a), \quad \hat{b} = b(1 + \delta_b)\]

where \(\delta_a\) and \(\delta_b\) are small relative errors on the order of \(\epsilon_{\text{mach}}\).

When you subtract \(\hat{b}\) from \(\hat{a}\), the leading significant digits that \(a\) and \(b\) share cancel out. The absolute error in the difference \(\hat{a} - \hat{b}\) remains roughly the same size as the absolute errors in \(\hat{a}\) and \(\hat{b}\):

\[(\hat{a} - \hat{b}) - (a - b) = a\delta_a - b\delta_b\]

However, because \(a \approx b\), the true difference \(a - b\) is much smaller than \(a\) or \(b\) individually. The relative error of the result is therefore:

\[\frac{|(\hat{a} - \hat{b}) - (a - b)|}{|a - b|} \approx \frac{|a\delta_a - b\delta_b|}{|a - b|}\]

Because the denominator \(|a - b|\) is very small, this relative error can grow enormously. The remaining bits in the significand are shifted to the left to normalize the result, and the vacated lower bits are filled with zeros or meaningless rounding noise.

A Concrete Decimal Example

To see how this happens, consider a simplified decimal floating-point system that stores only 5 significant digits. Suppose you want to compute \(x - y\) where:

\[x = 1.234567, \quad y = 1.234512\]

The true difference is:

\[x - y = 0.000055 = 5.5 \times 10^{-5}\]

In a 5-digit floating-point system, \(x\) and \(y\) are first rounded to:

\[\hat{x} = 1.2346, \quad \hat{y} = 1.2345\]

Subtracting the rounded values gives:

\[\hat{x} - \hat{y} = 0.0001 = 1.0 \times 10^{-4}\]

The computed result is \(1.0 \times 10^{-4}\) instead of \(5.5 \times 10^{-5}\), which is an error of over 81 percent. Four of the five significant digits were lost in the subtraction, and the remaining digit is dominated by the earlier rounding error.

Common Scenarios Where It Occurs

Catastrophic cancellation frequently appears in standard mathematical formulas if they are implemented naively:

The quadratic formula: When solving \(ax^2 + bx + c = 0\) with \(b^2 \gg 4ac\), the square root \(\sqrt{b^2 - 4ac}\) is nearly equal to \(|b|\). Computing \(-b + \sqrt{b^2 - 4ac}\) for positive \(b\) subtracts two nearly equal numbers.
Numerical differentiation: Approximating a derivative using \(\frac{f(x + h) - f(x)}{h}\) for very small step sizes \(h\) causes \(f(x + h)\) and \(f(x)\) to be nearly equal, destroying accuracy as \(h\) shrinks.
Variance calculation: Computing variance with the one-pass formula \(\frac{1}{n}\sum x_i^2 - \bar{x}^2\) subtracts two large, nearly equal sums when the data values have a large mean and small spread.

How to Avoid Catastrophic Cancellation

The most effective way to prevent catastrophic cancellation is to reformulate the mathematical expression so that nearly equal numbers are not subtracted directly.

For the quadratic formula when \(b > 0\), you can compute the root with the larger magnitude first:

\[x_1 = \frac{-b - \sqrt{b^2 - 4ac}}{2a}\]

Since both terms in the numerator have the same sign, no cancellation occurs. You can then find the second root using Vieta's formula:

\[x_2 = \frac{c}{a x_1}\]

For expressions like \(\sqrt{x + 1} - \sqrt{x}\) when \(x\) is large, multiplying by the conjugate eliminates the subtraction:

\[\sqrt{x + 1} - \sqrt{x} = \frac{1}{\sqrt{x + 1} + \sqrt{x}}\]

When algebraic reformulation is not possible, using higher-precision arithmetic—such as 64-bit doubles instead of 32-bit floats, or arbitrary-precision libraries—can provide enough extra significant digits to preserve accuracy after cancellation.