Pattern-Based File Searching in Python with glob

Python's built-in glob module provides an efficient, readable way to search file systems using Unix shell-style wildcards. This guide explains how glob matches filenames, handles pattern syntax, traverses complex directory trees recursively, and minimizes memory consumption when handling large datasets.

How glob Works

Under the hood, glob combines the standard os.scandir() (or os.listdir()) directory reading functions with the pattern-matching logic implemented in the fnmatch module. Unlike standard string matching, glob matches file paths against wildcard patterns before returning the results as strings.

Because it interacts directly with the underlying filesystem rather than loading entire directory structures into memory first, it provides a fast and lightweight alternative to manual path iteration.

Supported Pattern Syntax

The glob module relies on three primary wildcard operators:

• Asterisk (*): Matches zero or more characters within a single directory level. For example, *.py matches all Python files in the current folder, but will not look inside subfolders.
• Question Mark (?): Matches exactly one character. For example, image_?0.png matches image_10.png and image_20.png, but not image_100.png.
• Character Ranges ([...]): Matches any single character specified in the brackets. Ranges can be alphanumeric, such as [0-9] or [a-z]. For example, log_[0-9].txt matches single-digit logs. You can negate ranges with an exclamation mark, like [!0-9].

To escape special characters like * or ? that are literal parts of a file name, wrap them in brackets (e.g., [?] or [*]).

Recursive Searching with **

By default, glob restricts wildcard evaluation to single directory levels. To search through arbitrary depths in directory trees, set the recursive parameter to True and use the double-asterisk (**) pattern.

import glob

# Search for all .csv files across all subdirectories
csv_files = glob.glob('/path/to/data/**/*.csv', recursive=True)

When recursive=True is enabled:

• The ** segment matches zero, one, or multiple nested directories.
• Trailing separators handle root and leaf levels smoothly.
• Setting recursive=False (the default) causes Python to treat ** identically to a single *.

Memory Management with iglob

The standard glob.glob() function reads all matching file paths into memory and returns them as a Python list. If you are scanning extensive directory trees containing millions of files, this list can lead to significant memory overhead.

To avoid this, use glob.iglob(). It accepts the same arguments as glob.glob() but returns a generator that yields matching file paths lazily:

import glob

# Lazily iterate through matching files
for file_path in glob.iglob('/var/log/**/*.log', recursive=True):
    # Process each file individually without loading all paths into memory
    print(file_path)

Modern Alternative: pathlib.Path.glob

Starting in Python 3.4, the object-oriented pathlib module also includes glob() and rglob() methods. While the standalone glob module operates on standard string paths, pathlib returns Path objects:

from pathlib import Path

root = Path('/path/to/data')
# rglob automatically applies recursive search
for file in root.rglob('*.csv'):
    print(file.name, file.resolve())

Use the standard glob module when working with string-based path pipelines and legacy codebases, and use pathlib.Path.glob when building object-oriented path manipulation logic.