Py_INCREF and Py_DECREF in Python C Extensions

In Python C extensions, memory management relies directly on reference counting, and the Py_INCREF and Py_DECREF macros are the primary tools used to control this mechanism. This article explains the core purpose of these two macros, how they regulate Python object lifecycles, the difference between owned and borrowed references, and the critical practices required to prevent memory leaks and segmentation faults when interfacing C with Python.

The Role of Reference Counting

CPython manages memory automatically through reference counting supplemented by a cyclic garbage collector. Every Python object (PyObject) contains an internal counter (ob_refcnt) that tracks how many active references point to it.

When an object's reference count drops to zero, it is deemed unreachable, and Python immediately frees its allocated memory. In pure Python, the interpreter manages this count implicitly. In C extensions, developers must manipulate this count manually using Py_INCREF and Py_DECREF.

What `Py_INCREF` Does

The Py_INCREF(PyObject *o) macro increments the reference count of the target object by one.

You use Py_INCREF when:

You acquire a new pointer to an existing object and intend to hold onto it (e.g., storing it in a global variable, a C struct, or a container).
You convert a borrowed reference into an owned reference to guarantee the object will not be destroyed while your code is using it.

PyObject *my_list = PyList_New(0);
/* You now hold an owned reference to my_list */

Py_INCREF(item);
/* Incrementing 'item' ensures it stays alive while inside the list */
PyList_Append(my_list, item);

If the pointer might be NULL, use Py_XINCREF instead, which performs a null-check before incrementing.

What `Py_DECREF` Does

The Py_DECREF(PyObject *o) macro decrements the reference count of the target object by one.

If decrementing causes the reference count to reach zero, Py_DECREF invokes the object's type deallocator function (the tp_dealloc slot), deallocating the object's memory and decrementing the counts of any other objects it contains.

You use Py_DECREF when:

You are done using an object for which you hold an owned reference.
An operation fails mid-execution and you must clean up previously created objects before returning NULL to indicate an error.

PyObject *result = PyObject_CallObject(func, args);
/* Handle result */
Py_DECREF(result); /* Release ownership when finished */

If the pointer can potentially be NULL, use Py_XDECREF to avoid a crash.

Owned vs. Borrowed References

Understanding when to call these macros depends on whether a reference is owned or borrowed:

Owned Reference: Your code is responsible for the reference. You must eventually call Py_DECREF on it, pass ownership to another function/container, or return it to Python. Functions like PyLong_FromLong, PyDict_New, and PyObject_GetAttrString return owned references.
Borrowed Reference: You have access to the object, but someone else (such as a container or the caller) owns it. The object is only guaranteed to exist as long as the owner keeps it alive. Functions like PyTuple_GetItem and PyList_GetItem return borrowed references. If you need to retain a borrowed reference beyond the current function call, you must call Py_INCREF to take ownership.

Consequences of Mismanagement

Because C lacks memory safety guarantees, errors with these macros lead to severe issues:

Memory Leaks: Forgetting to call Py_DECREF on an owned reference prevents the reference count from ever reaching zero. The memory consumed by the object and its children is never reclaimed.
Dangling Pointers and Segmentation Faults: Calling Py_DECREF too many times drops the count to zero prematurely. Subsequent access to that pointer accesses freed memory, causing crashes or corrupted data.
Null Pointer Dereference: Passing a NULL pointer to Py_INCREF or Py_DECREF directly causes an immediate segmentation fault. Py_XINCREF and Py_XDECREF must be used whenever an operation might return NULL upon failure.