How Python Serializes Code with Marshal

This article explores how Python uses its internal marshal module to serialize executable code objects into raw binary streams. You will learn the mechanics behind converting compiled Python code into persistent bytecode, how marshal structures this binary data, the difference between marshal and pickle, and the critical limitations regarding cross-version compatibility and security.

What Is the marshal Module?

The marshal module is an internal Python module designed primarily to support Python's module-caching mechanism. When Python imports a script, it compiles the source code into bytecode and caches the result on disk as a .pyc file inside the __pycache__ directory. To write this bytecode to disk and read it back efficiently, Python serializes the in-memory code object using marshal.

Unlike general-purpose serialization tools like pickle or json, marshal is hardcoded to support only a specific subset of internal Python types: primitive values (integers, floats, strings, booleans, None), basic collections (tuples, lists, sets, dictionaries), and executable code objects.

Understanding the Code Object

Before serialization occurs, Python compiles source code into a code object (types.CodeType). A code object contains everything the Python Virtual Machine (PVM) needs to execute a block of logic:

• co_code: The raw bytecode instructions.
• co_consts: A tuple of literals and nested code objects referenced in the bytecode.
• co_names: Global names, attribute names, and imported modules.
• co_varnames: Local variable names.
• co_filename, co_name, co_firstlineno: Debugging metadata such as file name, scope name, and line numbers.

The Serialization Process (marshal.dumps)

When you invoke marshal.dumps(code_object), the Python C-API traverses the underlying C structure (PyCodeObject) recursively and converts it into a sequence of bytes.

1. Type Descriptors (Type Tags): Every serialized entity begins with a single byte representing its type tag, defined in Python's source as constants like TYPE_CODE, TYPE_STRING, TYPE_INT, or TYPE_SMALL_TUPLE.

2. Recursive Decomposition: When marshal encounters a code object, it writes the TYPE_CODE tag. It then sequentially writes the code object's internal fields (argument count, register count, stack size, flags, bytecode, constants, names, etc.). Because the co_consts field often contains other code objects (such as nested functions or class definitions), the serializer calls itself recursively on those sub-objects.

3. String and Reference Interning (The Flag System): Modern Python versions use a variation called FLAG_REF combined with type tags. If an object (like a variable name or string) appears multiple times within the code object, marshal assigns it an index and references it later rather than writing duplicate bytes, keeping .pyc files compact.

Example of serializing and inspecting a function's code:

import marshal

def hello():
    return "Hello, World!"

# Extract and serialize the code object
code = hello.__code__
serialized_data = marshal.dumps(code)

print(type(serialized_data))  # <class 'bytes'>
print(serialized_data[:10])   # Binary representation starting with type tags

Deserialization and Execution (marshal.loads)

Deserialization is the reverse process. When marshal.loads(bytes_data) is called:

1. The deserializer reads the initial byte to determine the type tag.
2. Upon recognizing a TYPE_CODE tag, it allocates a new PyCodeObject structure in memory.
3. It reconstructs each associated field by reading length prefixes and sub-components.
4. The resulting object is a fully functional in-memory code object that can be passed directly to Python's built-in exec() or eval() functions.

# Reconstructing the executable object
reconstructed_code = marshal.loads(serialized_data)

# Executing the deserialized bytecode
exec(reconstructed_code)

marshal vs. pickle

While both modules convert Python structures into bytes, their purposes diverge significantly:

• Type Support: pickle supports user-defined classes, instance states, and complex object graphs. marshal only serializes built-in primitives and code objects; it cannot serialize arbitrary class instances.
• Cross-Version Stability: The pickle protocol guarantees backward and forward compatibility across Python versions. The marshal format does not. The byte layout of code objects frequently changes between minor Python releases (e.g., between Python 3.11 and 3.12).
• Speed: marshal is written directly in C for maximum throughput, designed purely to reduce Python interpreter startup latency.

Security and Version Caveats

The marshal module is not designed to be secure against erroneous or maliciously crafted data. Deserializing an untrusted marshal payload can cause memory corruption, segmentation faults, or arbitrary code execution via forged bytecode instructions.

Furthermore, because bytecode specifications differ across Python versions, a code object marshaled in Python 3.10 will typically fail to load or execute in Python 3.11 or later. For persistent, long-term storage or data exchange between different systems, standard serialization formats should always be preferred over marshal.