Python C API: Borrowed vs Owned References

Managing memory in Python C extensions requires a solid understanding of reference counting, specifically the distinction between owned references and borrowed references. At the language and compiler level, both reference types share the exact same C type—a pointer to PyObject—meaning Python runtime does not technically differentiate between them automatically. Instead, the distinction is an API contract governed entirely by reference counting rules that the extension developer must manually enforce to prevent memory leaks and segmentation faults.

The Core Difference: Ownership and Lifetime

Every Python object contains an internal reference count (ob_refcnt). When a C extension interacts with an object, ownership dictates who holds responsibility for that count.

• Owned References: Holding an owned reference means your C code has an explicit claim on the object. You are responsible for disposing of this reference when it is no longer needed by calling Py_DECREF() or Py_XDECREF(). Failing to do so causes a memory leak. If you pass an owned reference to another function that takes over ownership, that process is called "stealing" a reference.
• Borrowed References: A borrowed reference provides a pointer to a PyObject* without incrementing its reference count. The caller does not own the reference and must not call Py_DECREF() on it. The validity of a borrowed reference depends entirely on the lifetime of the true owner (typically a container or internal frame).

How Python Implements the Distinction

Because both forms are represented as PyObject*, Python's runtime engine has no metadata flag on the pointer to differentiate between an owned and borrowed reference. The differentiation exists strictly through API design and conventions:

1. API Return Values: Python's C API documentation defines whether a function returns an owned or a borrowed reference.

   • Functions that create new objects (like PyLong_FromLong() or PyList_New()) or retrieve object attributes (like PyObject_GetAttrString()) return owned references. The caller owns the reference and must clean it up.
   • Functions that access elements inside collections without altering their state (such as PyTuple_GetItem() or PyList_GetItem()) return borrowed references. The container maintains ownership.

2. Reference Stealing: Some API functions actively consume (steal) an owned reference rather than requiring you to manage it after the call. A common example is PyList_SetItem(list, index, item), which assumes ownership of item. If the call succeeds, you must not call Py_DECREF(item).

Converting Borrowed References to Owned References

Borrowed references can introduce dangerous race conditions or dangling pointers if the owner releases the object while your C code is still using it. For example, modifying a list while holding a borrowed reference to one of its items might trigger garbage collection on that item.

To safely extend the lifetime of a borrowed reference, convert it into an owned reference by explicitly incrementing its reference count:

PyObject *borrowed_item = PyList_GetItem(my_list, 0); // Borrowed

/* Convert to an owned reference to prevent unexpected deallocation */
Py_INCREF(borrowed_item);

/* Perform operations that might modify my_list */
do_something_mutating(my_list);

/* Release ownership when finished */
Py_DECREF(borrowed_item);

Static Analysis and Tooling

Because the C compiler cannot identify incorrect reference handling at compile time, developers rely on documentation, code review, and specialized static analysis tools like cpychecker or the internal debug builds of Python (./configure --with-pydebug). Debug builds track active references and can detect imbalances, helping verify that borrowed references are never decremented and owned references are never abandoned.