How Python Compiles Source Code into PYC Bytecode

When Python runs a script, it automatically translates human-readable source code into an intermediate format known as bytecode, which is cached in .pyc files within a __pycache__ directory. This article covers how Python parses source code into an Abstract Syntax Tree (AST), generates bytecode, serializes it using the internal marshal format, and creates .pyc files to accelerate future program startups.

1. Parsing Source Code into an Abstract Syntax Tree

Before producing bytecode, Python must analyze the syntax of your .py file. This process occurs in two primary stages:

1. Tokenization: The lexical analyzer scans the raw text of the Python source file and breaks it down into a stream of tokens, such as keywords, identifiers, operators, and literals.
2. Parsing: The parser processes the token stream and verifies that it adheres to Python's grammatical rules. It then constructs an Abstract Syntax Tree (AST), a hierarchical tree representation of the code's logical structure.

2. Emitting Bytecode

Once the AST is constructed, Python's compiler transforms the tree into bytecode. Bytecode consists of compact, platform-independent numeric instructions designed for execution by the Python Virtual Machine (PVM).

Each bytecode instruction corresponds to an opcode (operation code), such as LOAD_CONST, STORE_FAST, or BINARY_OP, often followed by an argument. Python bundles these instructions into a code object. A code object contains the raw bytecode along with essential execution metadata, including variable names, constant values, line number mappings, and required stack depth.

3. Structure of a .pyc File

To avoid recompiling the source code on subsequent runs, Python writes the code object to disk as a .pyc file. A modern .pyc file contains four distinct components in its binary header followed by the payload:

• Magic Number (4 bytes): A unique identifier representing the Python interpreter version. If the interpreter changes, Python knows the cached bytecode is incompatible.
• Bit Field / Flags (4 bytes): Specifies how cache validation is handled (e.g., whether it relies on timestamps or cryptographic hashes).
• Validation Data (8 bytes):
  • In timestamp-based invalidation (default), this holds the 4-byte modification timestamp of the source .py file, followed by the 4-byte source file size.
  • In hash-based invalidation (PEP 552), this stores a 64-bit cryptographic hash of the source code.
• Marshalled Code Object: The serialized representation of the compiled code object, generated using Python’s internal marshal module.

4. Writing to the __pycache__ Directory

Python does not place .pyc files directly alongside .py files. Instead, it stores them in a subdirectory named __pycache__.

The compiled file follows a standardized naming convention: <module_name>.<interpreter-tag>.pyc (for example, utils.cpython-311.pyc). This format ensures that multiple Python versions and implementations can run in the same environment without overwriting each other's cached files.

Python generally writes .pyc files only for imported modules, not for the top-level script executed directly from the command line, unless explicitly instructed via command-line flags or the py_compile module.

5. Cache Validation and Recompilation

When Python imports a module, it checks the __pycache__ folder to see if a matching .pyc file already exists.

If found, Python reads the header:

• For timestamp-based validation, it compares the timestamp and file size in the .pyc header against the filesystem metadata of the .py source file.
• For hash-based validation, it compares the stored hash against the hash of the current source file.

If the validation data matches, Python skips the compilation step entirely, unmarshals the cached code object directly into memory, and passes it to the PVM for execution. If the source file is newer, modified, or missing a compiled counterpart, Python recompiles the source code and updates the .pyc file.