Python C API Py_buffer Struct Fields Explained

Python's buffer protocol provides a standardized C-level interface for zero-copy data sharing between Python objects, which is critical for high-performance computing, image processing, and data manipulation libraries like NumPy. At the center of this protocol is the Py_buffer struct, a data structure that describes the underlying layout, format, and properties of an object's memory. This article breaks down all the fields defined in the Py_buffer struct and explains their specific roles in exposing and interacting with native memory buffers.

The Py_buffer Struct Definition

Defined in Python's header files (typically <Python.h> or <bufferobject.h>), the Py_buffer struct consists of the following standard fields:

typedef struct bufferinfo {
    void *buf;
    PyObject *obj;
    Py_ssize_t len;
    Py_ssize_t itemsize;
    int readonly;
    int ndim;
    char *format;
    Py_ssize_t *shape;
    Py_ssize_t *strides;
    Py_ssize_t *suboffsets;
    void *internal;
} Py_buffer;

Core Data and Buffer Size Fields

void *buf

The buf field is a generic pointer to the logical beginning of the memory block being exposed. Depending on how the strides are configured, this does not necessarily point to the lowest physical memory address of the allocated block, but rather the element at index (0, 0, ..., 0).

Py_ssize_t len

This field contains the total length of the logical memory block in bytes. It represents the total size that the consumer can access, calculated generally as itemsize multiplied by the total number of elements.

int readonly

An integer acting as a boolean flag. A value of 1 indicates that the buffer is read-only and attempts to write to buf should be prevented. A value of 0 indicates that the buffer supports read and write operations.

Element and Type Metadata

Py_ssize_t itemsize

This field stores the storage size of a single element in bytes. For an array of standard 32-bit signed integers, itemsize would be 4.

char *format

A NUL-terminated C string that describes the data type of each individual element using Python's struct module syntax (PEP 3118 format strings). For example:

• "B" indicates an unsigned byte.
• "i" indicates a standard signed integer.
• "d" indicates a double-precision float.
• NULL is implicitly interpreted as "B" (unsigned bytes).

Multidimensional Layout Fields

int ndim

The number of dimensions (axes) represented by the buffer. A flat block of memory has an ndim of 1, whereas a 2D matrix has an ndim of 2. A scalar has an ndim of 0.

Py_ssize_t *shape

An array of ndim integers defining the size of the buffer along each dimension. For example, a 3x4 matrix would have a shape array containing {3, 4}. If ndim is 0, this field should be NULL.

Py_ssize_t *strides

An array of ndim integers representing the number of bytes needed to jump to the next element along each corresponding dimension.

• In a standard row-major (C-style) contiguous array, strides[ndim - 1] equals itemsize.
• In a column-major (Fortran-style) array, strides[0] equals itemsize. If strides is NULL, the buffer is assumed to be C-contiguous.

Py_ssize_t *suboffsets

An array of ndim integers used for arrays of pointers (non-contiguous, nested slices), common in languages like C when handling multidimensional arrays defined as type**.

• A value >= 0 at index i specifies the byte offset to add to the pointer dereferenced at that dimension.
• A negative value indicates that no pointer dereferencing is required for that dimension (standard strided indexing). If the buffer contains a simple flat memory block without pointer dereferencing, suboffsets must be NULL.

Ownership and Lifecycle Fields

PyObject *obj

A pointer to the exporting Python object that owns the underlying memory. When a consumer requests a buffer using functions like PyObject_GetBuffer(), this field stores a new reference to the source object (Py_INCREF is called). When the consumer is finished with the buffer, passing the struct to PyBuffer_Release() decrements this reference to ensure safe garbage collection.

void *internal

A private pointer reserved for the exporting object. The consumer must never modify or inspect this field. It is often used by the exporting object to track internal state, dynamic allocations for shape or strides, or custom memory managers that need cleanup upon release.