How RustPython Compiles and Interprets Python

RustPython interprets Python syntax entirely within Rust by leveraging a modular, multi-stage architecture that cleanly separates lexical analysis, Abstract Syntax Tree (AST) generation, bytecode compilation, and virtual machine execution. By decoupling source parsing from runtime execution into distinct Rust crates, the interpreter processes standard Python source code into an intermediate bytecode format before executing it on a custom, stack-based Rust virtual machine without relying on the CPython C runtime.

The Modular Crate Architecture

The core architecture of RustPython is partitioned into specialized crates, each handling an isolated stage of the compilation and interpretation pipeline:

• rustpython-parser: Implements the lexer and parser. It processes raw Python source code strings, performing lexical analysis to generate tokens, which are then parsed into a structured AST.
• rustpython-ast: Defines the data structures representing Python syntax nodes. This crate models language constructs (statements, expressions, definitions) as Rust enums and structs.
• rustpython-compiler: Traverses the AST and emits executable bytecode. It handles symbol resolution, scope analysis, and the emission of intermediate instructions packaged into code objects.
• rustpython-vm: Contains the virtual machine runtime that executes compiled bytecode, manages the call stack, handles runtime exceptions, and provides the Python dynamic object model.

1. Lexing and AST Generation

The process begins when raw Python text enters the parser. RustPython analyzes the source and transforms it into an AST representation. Because Rust’s type system features exhaustive pattern matching via enums, the AST cleanly encapsulates all legal Python language structures. Every statement (e.g., If, FunctionDef, Assign) and expression (e.g., BinOp, Call, Constant) maps directly to strongly typed Rust variants.

2. Bytecode Compilation

Once the AST is validated, the compiler crate transforms these high-level nodes into a flat instruction set. Python source is not translated into native machine code (such as LLVM IR); instead, it compiles into custom RustPython bytecode instructions.

During this pass, the compiler:

• Resolves local, closure, and global variable scopes.
• Constructs symbol tables.
• Flattens structured control-flow logic (loops, conditionals) into sequential opcodes combined with jump instructions.
• Packages the resulting instructions, variable names, and constants into a CodeObject, matching the operational model of traditional Python compilation.

3. The Stack-Based Virtual Machine

The rustpython-vm crate takes the compiled CodeObject and executes it inside an evaluation loop. RustPython uses a stack-based virtual machine model where operations push and pop operands onto an execution frame stack.

The execution engine reads opcodes sequentially:

• Arithmetic opcodes fetch values from the frame stack, evaluate them, and push results.
• Control flow opcodes manipulate the instruction pointer based on boolean evaluation.
• Function calls allocate a new execution frame with an isolated scope and stack.

Dynamic Typing and Memory Architecture

To replicate Python's dynamic type system inside Rust's static memory model, RustPython uses reference-counted pointer abstractions, primarily PyObjectRef (wrapping an Arc or Rc).

Python objects are represented as Rust heap allocations containing a payload implementing the PyPayload trait alongside a reference to their associated PyType. Dynamic method dispatch, duck typing, and attribute lookups are mediated through Rust traits, mapping Python methods (__add__, __init__, __getattr__) to Rust functions. Memory management relies on Rust’s built-in ownership semantics combined with interior mutability, enabling Python's garbage collection and reference dynamics without interfacing with C code.