CPython C-API: tp_alloc, tp_new, and tp_init Explained

Creating custom extension types in the CPython C-API requires managing how instances are allocated in memory, constructed, and initialized. When a Python class is instantiated, CPython delegates these responsibilities across three dedicated slots in the PyTypeObject struct: tp_alloc, tp_new, and tp_init. This article explains the exact purpose of each slot, how they interact, and their roles in the object creation lifecycle.

The Object Lifecycle Overview

When code calls a type like obj = MyType(arg1, key=val), CPython's default type call handler (type_call) executes a three-phase sequence:

1. tp_new is invoked to construct and return an instance of the class.
2. Inside tp_new, tp_alloc is typically called to reserve heap memory for that instance.
3. If tp_new successfully returns an instance of MyType (or a subclass), CPython automatically calls tp_init to initialize the instance's state.

1. tp_alloc: Memory Allocation

PyObject* (*allocfunc)(PyTypeObject *type, Py_ssize_t nitems);

Purpose

tp_alloc handles the low-level allocation of raw memory required to store an instance of the type.

Responsibilities

• Memory Allocation: It allocates the correct amount of memory based on type->tp_basicsize (and type->tp_itemsize for variable-length objects like tuples or strings).
• Header Setup: It initializes the object's ob_refcnt to 1 and sets its ob_type pointer to the target type.
• Zero-Filling: It typically zeroes out the rest of the allocated memory block to ensure pointers start in a clean NULL state.

Usage

Most custom extension types do not need to implement their own tp_alloc. Instead, they inherit or explicitly set PyType_GenericAlloc. Custom implementations are only necessary if an extension requires specialized memory pools, arena allocators, or unique alignment constraints.

2. tp_new: Instance Construction

PyObject* (*newfunc)(PyTypeObject *subtype, PyObject *args, PyObject *kwds);

Purpose

tp_new represents the constructor of the class and corresponds directly to Python's __new__() method. Its primary role is to produce and return an instance of the type.

Responsibilities

• Triggering Allocation: It typically calls subtype->tp_alloc(subtype, nitems) to obtain uninitialized memory for the object.
• Immutable Field Setup: Because tp_new controls the initial creation, any C-level pointers, structures, or immutable data that must exist before initialization should be configured here.
• Subtype Handling: When inherited, tp_new receives the final subtype being instantiated, allowing base classes to allocate memory with the derived type's layout.
• Instance Control: It can return an existing instance instead of creating a new one (e.g., implementing singletons, interned values, or flyweight patterns).

Return Value

• Returns a new reference to the created PyObject*.
• Returns NULL if an error occurs (setting a Python exception).

3. tp_init: Instance Initialization

int (*initproc)(PyObject *self, PyObject *args, PyObject *kwds);

Purpose

tp_init represents the initializer of the class and corresponds directly to Python's __init__() method. It receives the already-allocated instance and configures its mutable state.

Responsibilities

• Argument Parsing: It parses positional (args) and keyword (kwds) arguments passed to the class constructor, typically using PyArg_ParseTupleAndKeywords.
• State Configuration: It assigns initial values to instance attributes and populates C-level fields inside the struct.
• Re-initialization Safety: Because Python permits calling obj.__init__() multiple times explicitly on an existing object, tp_init must cleanly release or replace previous state to prevent memory leaks.

Return Value

• Returns 0 on success.
• Returns -1 on error (setting a Python exception).

Summary of Differences