Safe Python Refactoring with Bowler and CSTs

Automating large-scale refactorings in Python often introduces the risk of broken syntax, destroyed code formatting, or stripped comments. Bowler resolves these issues by using Concrete Syntax Trees (CSTs) to inspect and mutate source code while preserving its exact physical structure, including whitespace, indentation, and docstrings. This article explains how Bowler utilizes CSTs to target, transform, and safely apply automated modifications across extensive Python codebases without altering untouched code style.

Abstract Syntax Trees vs. Concrete Syntax Trees

To understand Bowler’s safety model, one must contrast Abstract Syntax Trees (ASTs) with Concrete Syntax Trees (CSTs).

Python’s built-in ast module compiles code into an abstract structural representation optimized for the Python runtime. In an AST, elements irrelevant to execution—such as comments, exact whitespace formatting, parenthesis groupings, and newlines—are discarded. While ASTs are useful for static analysis or linting, writing an AST back to disk requires a code generator. This re-serialization process inevitably destroys existing formatting and strips comments.

A Concrete Syntax Tree, by contrast, retains a 1:1 mapping with the raw source file. Every comma, space, indentation block, newline, and inline comment is captured as a distinct node or metadata attribute. Bowler operates directly on this full-fidelity representation. By modifying only the targeted CST nodes, Bowler guarantees that any untouched code remains completely identical down to the byte level.

The Underlying Engine: CST Parsing with Fissix

Bowler is built on top of fissix, a maintained fork of Python's lib2to3 library. When Bowler processes a Python file, the engine executes the following steps:

1. Tokenization and Parsing: The source text is parsed into a tree consisting of Node and Leaf instances.
2. Prefix Preservation: In fissix, each Leaf contains a prefix attribute. This attribute stores any whitespace, indentation, or comments that appeared immediately before that token.
3. In-Place Mutation: When a refactoring rule triggers, Bowler alters, inserts, or removes specific Node or Leaf objects. It reallocates the necessary prefix strings to maintain syntactical correctness and proper visual alignment.
4. Serialization: Because the CST contains every character from the original file, serialization simply involves traversing the tree and writing the tokens and their prefixes back to disk.

The Query and Transformation Lifecycle

Bowler abstracts raw CST operations behind a fluent, chainable API designed for querying and modifying code. A typical Bowler execution follows a three-stage lifecycle:

1. Selection

Bowler provides high-level selectors to locate specific code patterns without requiring manual tree-traversal algorithms. Developers can select language constructs using patterns such as:

• Function definitions (.select_function("name"))
• Class definitions (.select_class("name"))
• Function calls and argument lists (.select_call("name"))
• Custom pattern strings matching grammar nodes

2. Transformation

Once Bowler locates a matching CST node, it applies a transformation. Transformations can be high-level convenience methods (such as .rename("new_name") or .add_argument(...)) or custom callback functions.

When using a callback, Bowler passes the matched Node directly to the developer. The developer can inspect child leaves, update attributes, or replace the node with new CST structures generated dynamically from code snippets.

3. Execution and Diff Verification

Bowler does not apply changes blindly. It generates unified diffs for every change, allowing developers to inspect the exact textual output before writing modifications to disk. It supports:

• Interactive Mode: Prompting the user to accept, reject, or skip individual file changes.
• Write Mode: Applying changes directly once verified.
• Silent/Dry-Run Mode: Generating diffs for continuous integration (CI) environments to check if code adheres to desired patterns.

Why Bowler Ensures Codebase Safety

Bowler’s combination of CST manipulation and execution safety mechanisms provides distinct advantages over regex-based or AST-based approaches:

• Zero Formatting Regressions: Unlike code formatters that reformat entire files, Bowler touches only the target nodes. The rest of the file keeps its historical git blame and original formatting intact.
• Syntactic Awareness: Unlike string replacement or regular expressions, Bowler understands variable scopes, call sites, and definitions. It will not rename a local variable that happens to share a name with a selected global function.
• Deterministic Output: Because CSTs accurately map Python grammar, transformations either succeed cleanly or fail explicitly during the parse phase, preventing partial or corrupted syntax writes.

By leveraging Concrete Syntax Trees through a declarative API, Bowler transforms complex architectural migrations into predictable, safe, and automated operations.