Python glob.iglob vs glob.glob for Large Directories

When searching through large filesystems in Python, choosing between glob.glob() and glob.iglob() directly impacts your application's performance and resource consumption. While glob.glob() loads all matching paths into memory at once as a standard list, glob.iglob() returns a generator that yields results lazily, one file at a time. This article explores how lazy evaluation reduces memory consumption, decreases time-to-first-result, and enables efficient early-exit operations across large file trees.

Memory Optimization: Lists vs. Generators

The primary difference between the two functions lies in how they allocate memory.

glob.glob() evaluates the filesystem traversal eagerly. It discovers every matching path across the target directory, constructs string objects for each path, and populates a Python list before returning. On a filesystem with hundreds of thousands or millions of files, this behavior causes a significant memory spike:

import glob

# Loads all matching paths into a single in-memory list (High RAM usage)
all_files = glob.glob("/large_dataset/**/*.csv", recursive=True)

In contrast, glob.iglob() operates as an iterator. It yields each file path individually as it encounters it on the disk. Because it does not store the entire collection of paths in memory, its memory footprint remains constant—\(O(1)\) space complexity—regardless of whether the directory contains ten files or ten million files:

import glob

# Yields paths one by one without creating a massive list (O(1) memory)
file_stream = glob.iglob("/large_dataset/**/*.csv", recursive=True)
for file_path in file_stream:
    process(file_path)

Decreased Latency (Time-to-First-Result)

Eager evaluation blocks execution until the entire directory tree has been indexed. If you run glob.glob() on a network-attached storage (NAS) device or a deeply nested directory, your script will pause until the entire scan finishes.

glob.iglob() yields the first match almost instantaneously. Your application can immediately start reading, parsing, or transferring files while the generator continues scanning the remaining directories in the background. This concurrency of search and processing dramatically reduces total job completion time.

Efficient Short-Circuiting and Early Exits

Many file-search operations do not require a complete scan of the filesystem. For instance, you might only need to find the first matching configuration file, inspect a batch of sample records, or verify whether any files match a specific pattern.

With glob.glob(), breaking out of a loop does not prevent the initial full scan:

# Inefficient: Scans the entire drive before the loop even starts
for file_path in glob.glob("/large_dataset/**/*.json", recursive=True):
    if is_valid(file_path):
        break

With glob.iglob(), breaking out of the loop immediately halts filesystem traversal, saving significant disk I/O and CPU cycles:

# Efficient: Halts directory traversal the moment a match is found
for file_path in glob.iglob("/large_dataset/**/*.json", recursive=True):
    if is_valid(file_path):
        break

When to Use Each Function

Use glob.glob() only when you require the complete list of files upfront, need to know the total file count before processing, or need to perform indexing and slicing operations (e.g., files[-10:]).

Use glob.iglob() by default for: