Python timeit: How Garbage Collection Skews Benchmarks

Micro-benchmarking code using Python’s timeit module requires controlled conditions to obtain reproducible results, but leaving Python's automatic garbage collection active can severely distort these measurements. This article explains how the generational garbage collector introduces non-deterministic latency spikes, why memory-allocating code is disproportionately impacted during benchmark runs, and why timeit disables garbage collection by default to ensure fair, microsecond-level code comparisons.

The Dual Nature of Python Memory Management

To understand why garbage collection (GC) skews micro-benchmarks, it is essential to distinguish between Python's two memory reclamation mechanisms:

1. Reference Counting: Python immediately deallocates an object when its reference count drops to zero. This overhead is deterministic and is directly paid by the code creating and discarding the object.
2. Cyclic Garbage Collector: Reference counting cannot clean up circular references (e.g., Object A references Object B, which references Object A). To resolve this, Python runs a background generational garbage collector (handling Generations 0, 1, and 2) that periodically freezes execution to detect and clean cyclic isolates.

Why Active Garbage Collection Distorts Benchmarks

1. Non-Deterministic Latency Spikes

The cyclic garbage collector runs only when the number of allocations exceeds a specific threshold relative to deallocations. In a micro-benchmark executing thousands of iterations, a GC sweep might trigger on iteration 4,500 and again on iteration 9,000, while the remaining iterations run completely uninterrupted.

Because timeit aggregates total run time across many loops, a single collection cycle—especially a Generation 2 collection inspecting long-lived objects—adds an unpredictable block of latency. Repeating the exact same test can yield wildly different results depending on whether an extra GC pass was triggered.

2. Contamination from External Objects

When the cyclic collector activates, it does not scan only the objects created inside the benchmark loop. It evaluates candidate objects across the entire Python runtime. If setup code, imported libraries, or test fixtures have allocated objects that reside in memory, the benchmarked code ends up paying the performance penalty for scanning and traversing those external objects.

3. Asymmetric Penalization of Allocating Code

Consider comparing two functions where Function A creates small, short-lived circular structures (like exceptions or linked nodes) and Function B performs pure arithmetic. If GC is enabled:

• Function A will trigger periodic collection sweeps, causing its reported time to reflect both its raw execution logic and the expensive runtime-wide GC sweeps.
• Function B will run uninterrupted.

While this difference reflects real-world behavior to an extent, it obscures the actual algorithmic execution speed of Function A by intertwining it with the arbitrary threshold mechanics of Python's collector.

The timeit Default: Isolated Execution

To solve this problem, timeit.timeit() explicitly disables the cyclic garbage collector by running gc.disable() prior to benchmarking. By disabling the collector:

• Every run operates under strictly identical conditions.
• The test measures only the deterministic execution path and basic reference-counting overhead.
• Outliers caused by multi-millisecond sweep phases are removed.

Balancing Micro-benchmarking and Production Reality

Disabling garbage collection creates a clean room environment, which is the primary goal of a micro-benchmark. However, developers must recognize the trade-off:

• When to keep GC disabled (Default): Leave GC disabled when isolating the CPU efficiency of specific functions, comparing alternative algorithms, or analyzing low-level syntax performance.
• When to enable GC (setup="gc.enable()"): Enable GC only when explicitly profiling memory pressure or testing whether an implementation creates so many circular references that GC overhead degrades overall system performance. For full-system performance analysis, standard profiling tools like cProfile or memory profilers provide more meaningful context than timeit.