Why Benchmarks Use Geometric Mean Over Arithmetic Mean
When evaluating computer hardware, reviewers run multiple tests and combine the results into a single overall score. While the arithmetic mean (the standard average) is common in everyday life, it severely distorts data when applied to diverse hardware benchmarks. This article explains why the geometric mean is the industry standard for aggregating benchmark scores, highlighting its ability to neutralize scale differences, accurately average performance ratios, maintain baseline independence, and prevent outlier distortion.
Eliminating Scale Dominance
Computer benchmarks measure vastly different workloads using different scales. One test might output a score of 15,000 points, while a gaming test outputs 60 frames per second (FPS).
The arithmetic mean is calculated by adding values together and dividing by the total number of tests. If you combine 15,000 and 60 using the arithmetic mean, the first test completely dominates the final score. A mere 5% improvement in the first test adds 750 to the total, whereas doubling the performance (a 100% improvement) in the second test only adds 60. The 60 FPS test becomes mathematically irrelevant.
The geometric mean solves this by multiplying the numbers together and taking the n-th root (where n is the number of tests). Because it relies on multiplication rather than addition, the geometric mean evaluates relative percentage changes rather than raw numerical differences. A 10% improvement in a 60 FPS test carries the exact same weight as a 10% improvement in a 15,000-point test.
Accurately Averaging Normalized Ratios
To bypass the issue of different scales, hardware reviewers often normalize results by comparing them to a baseline system, turning the scores into ratios or percentages. However, the arithmetic mean is mathematically flawed when averaging ratios.
Consider two processors, CPU A and CPU B, tested across two benchmarks:
- Test 1: CPU A is 2 times as fast as CPU B (Ratio of 2.0 to 1.0).
- Test 2: CPU A is half as fast as CPU B (Ratio of 0.5 to 1.0).
Logically, the two CPUs have equal overall performance, as their wins and losses are exact inverses.
If we use the arithmetic mean to average CPU A's normalized scores (2.0 and 0.5), we get: (2.0 + 0.5) / 2 = 1.25
The arithmetic mean falsely concludes that CPU A is 25% faster than CPU B overall. Worse, if we flip the baseline and normalize the scores to CPU A, the arithmetic mean would conclude that CPU B is 25% faster.
If we use the geometric mean, we multiply the scores and take the square root: Square root of (2.0 × 0.5) = Square root of 1.0 = 1.0
The geometric mean correctly identifies that the two processors are equal in overall performance, matching the reality of the data.
Independence from the Reference System
Because the arithmetic mean fails at averaging ratios, the final ranking of hardware in a review using the arithmetic mean can actually change depending on which computer is chosen as the baseline reference.
The geometric mean possesses a property known as baseline invariance. When using the geometric mean, it does not matter which piece of hardware is used as the 1.0 reference point. You can normalize the data against a ten-year-old budget processor or a brand-new flagship, and the relative percentage difference between all the tested components will remain exactly the same.
Dampening the Impact of Outliers
Computer architectures occasionally have specific quirks—such as a massive amount of cache or a specialized media engine—that allow them to achieve an anomalously high score in one specific benchmark while performing poorly in others.
The arithmetic mean allows a single, massive outlier to drag the overall average up, making a specialized or flawed piece of hardware look like a great all-rounder. The geometric mean naturally dampens the effect of high outliers. To achieve a high geometric mean, a processor or graphics card cannot rely on a single fluke result; it must perform consistently well across the entire suite of benchmarks.