How pytest-xdist Parallelizes Tests Across CPU Cores

The pytest-xdist plugin accelerates test suites by distributing individual tests across multiple CPU cores as separate worker processes. Instead of running tests sequentially within a single Python thread, pytest-xdist establishes a controller-worker architecture that collects test definitions, divides the workload using customizable scheduling algorithms, executes tests independently in parallel, and collates the final results into a unified test report.

The Controller-Worker Architecture

When you invoke pytest -n <NUM>, pytest-xdist designates the initial process as the controller (formerly referred to as the master) and spawns <NUM> separate worker processes. These worker processes are completely isolated Python interpreters managed via execnet, a Python library designed for secure, low-overhead inter-process communication.

Because Python's Global Interpreter Lock (GIL) limits true multi-threaded CPU concurrency in standard CPython, launching individual OS-level processes bypasses the GIL entirely. Each worker gets its own memory space and Python runtime, allowing simultaneous CPU utilization across all allocated cores.

Test Collection and Synchronization

Before execution begins, the controller and all worker processes perform the test collection phase:

1. Collection on Workers: Every worker process independently scans the codebase and collects the full list of test items.
2. Sanity Check: The workers report their collected item IDs back to the controller. The controller verifies that all workers collected the identical set of tests in the same order. This step guarantees deterministic indexing and prevents environment desynchronization across processes.

Workload Distribution Strategies

Once test collection is validated, the controller serves as a central dispatcher. It determines which worker runs which test based on the --dist strategy provided:

• --dist=load (Default): The controller maintains a pending queue of tests. It initially sends a batch of tests to each worker. As soon as a worker completes an item and becomes idle, the controller sends it the next pending test. This work-stealing approach optimizes resource efficiency when tests vary significantly in runtime.
• --dist=loadscope: Tests are grouped by module (file) or class scope. All tests within the same class or module are routed to the same worker process. This is critical for suites that rely on expensive class- or module-scoped fixtures, ensuring fixtures are set up and torn down only once per worker.
• --dist=loadfile: Similar to loadscope, but groups strictly at the file level regardless of classes.
• --dist=loadgroup: Allows manual control using the @pytest.mark.xdist_group(name="group_name") marker to force specific tests into the same worker process, useful for preventing race conditions on shared external resources.

Inter-Process Communication and Result Aggregation

During test execution, workers execute tests locally within their own process environments. As tests finish:

1. Serialization: Workers capture the execution status (Pass, Fail, Skip, XFail), execution times, stdout/stderr streams, and traceback data. This information is serialized into lightweight data structures.
2. Reporting via IPC: Workers stream the serialized reports back to the controller process over standard OS pipes managed by execnet.
3. Display Aggregation: The controller receives these reports and feeds them into Pytest's standard reporting hooks (pytest_runtest_logreport). To the user, terminal outputs and progress indicators appear as if the tests were running through standard Pytest execution.

Once all tests in the queue have run and their reports have been processed, the controller instructs the workers to shut down and prints the standard session summary.