How Python Closures Work: Cell Objects and closure

This article explains the internal mechanics of closures in Python, detailing how the runtime captures free variables from an enclosing scope. You will learn how Python uses specialized cell objects to store references to these lexical variables, how the inner function maintains access to them via its __closure__ attribute, and how the underlying bytecode coordinates this behavior to preserve state across independent function calls.

Understanding the Closure Mechanism

A closure occurs when a nested function references a variable defined in its enclosing scope, and that nested function is returned or passed elsewhere, outliving the scope in which it was created. Because standard local variables are destroyed when their enclosing function terminates, Python needs a way to keep referenced variables alive without turning them into globals.

Instead of binding the variable's value directly to the inner function at definition time, Python binds a reference to the storage location of the variable. This is accomplished using cell objects.

What Are Cell Objects?

A cell object (PyCellObject in CPython) is an internal container used to implement closures. It holds a pointer to a Python object rather than holding the object directly.

When a variable in an outer function is referenced by an inner function:

1. Python recognizes that the variable must survive beyond the execution frame of the outer function.
2. The outer function does not store the variable purely as a standard local in its stack frame. Instead, it creates a cell object to wrap the value.
3. Both the outer function and the inner function share a reference to this exact same cell object.

Because both scopes reference the same cell, any updates to the variable (such as those made with the nonlocal keyword) are immediately visible to both the outer and inner functions.

Inspecting Closures with __closure__

Every Python function has a __closure__ attribute. For standard functions that do not capture variables from an outer scope, __closure__ evaluates to None. For closures, __closure__ contains a tuple of cell objects corresponding to the captured variables.

Consider the following example:

def make_multiplier(factor):
    def multiply(number):
        return number * factor
    return multiply

doubler = make_multiplier(2)

Inspecting the doubler function reveals how Python encapsulates factor:

print(doubler.__closure__)
# Output: (<cell at 0x...: int object at 0x...>,)

cell = doubler.__closure__[0]
print(type(cell))
# Output: <class 'cell'>

print(cell.cell_contents)
# Output: 2

The cell_contents attribute provides direct access to the encapsulated variable inside the cell.

Bytecode and Code Object Attributes

The association between the outer scope and the inner function is governed by metadata in each function's code object (__code__):

• co_cellvars: Present on the enclosing function's code object, this tuple lists the names of local variables that are referenced by nested functions and thus need to be stored inside cell objects.
• co_freevars: Present on the inner function's code object, this tuple lists the names of variables used locally but defined in an enclosing scope.

print(make_multiplier.__code__.co_cellvars)
# Output: ('factor',)

print(doubler.__code__.co_freevars)
# Output: ('factor',)

When Python compiles the bytecode for these functions:

• The outer function uses instructions like MAKE_CELL or creates cell objects during frame initialization.
• The inner function accesses captured variables using LOAD_DEREF and modifies them using STORE_DEREF instead of standard LOAD_FAST and STORE_FAST instructions.

LOAD_DEREF retrieves the value by dereferencing the cell found in __closure__ at the specified index, allowing the function to execute seamlessly even after the outer function's execution frame has been popped off the call stack.