Python Module Search Paths: site.py and site-packages

This article provides an overview of how the Python runtime constructs its module search path (sys.path) during startup. It details the automatic execution of the standard library module site.py, the discovery and integration of system and user-level site-packages directories, and the role of .pth path configuration files in extending module resolution.

The Role of site.py During Initialization

When the Python interpreter initializes, it establishes a list of strings called sys.path that specifies the search path for modules. While core built-in modules and the standard library are registered early in the startup sequence, Python relies on the standard library module site.py to handle third-party libraries.

Unless the interpreter is launched with the -S command-line flag, site.py is imported automatically. Upon execution, it triggers a series of functions—primarily main()—responsible for determining installation prefixes, calculating directory locations for external packages, processing custom path configurations, and appending these locations to sys.path.

Discovering site-packages

The primary objective of site.py is finding and registering site-packages directories based on the interpreter's installation prefixes (sys.prefix and sys.exec_prefix).

1. Global Site Directories: On Unix-like systems, site.py constructs paths typically structured as lib/pythonX.Y/site-packages. On Windows, the default target is Lib\site-packages. These directories are validated for existence and appended to sys.path.
2. User-Specific Site Directories: Standardized under PEP 370, site.py also checks for per-user installations. If site.ENABLE_USER_SITE is true (and the environment variable PYTHONNOUSERSITE is not set), it appends user-level directories (such as ~/.local/lib/pythonX.Y/site-packages on Linux or %APPDATA%\Python\PythonXY\site-packages on Windows) to sys.path.

Processing .pth Files

While traversing each discovered site-packages directory, site.py inspects the directory contents for files ending with the .pth extension. These path configuration files allow packages to add additional directories to the search path without manually modifying environment variables.

For every .pth file found:

• Blank lines and lines beginning with # are ignored.
• Standard directory paths listed on individual lines are validated and added to sys.path.
• Lines beginning with import are executed directly as Python code, a mechanism often used by editable package installs (such as those generated via pip install -e) to dynamically configure import loaders.

Virtual Environments and Isolation

Virtual environments leverage this mechanism to isolate project dependencies. A virtual environment contains a configuration file named pyvenv.cfg in its root directory. When Python starts, it detects this file and alters sys.prefix to point to the virtual environment rather than the global installation.

Consequently, when site.py executes, it derives the site-packages location from the virtual environment's prefix, ensuring that the isolated environment's dependencies take precedence and system-wide packages are excluded by default.