NumPy Structured Arrays: Guide to C-Style Records

NumPy structured arrays allow developers to group heterogeneous data types into named fields within a single array, mirroring the behavior and memory layout of C structures (struct). This article explains how structured arrays work, how to define custom data types with multiple fields, and how their memory alignment enables fast, low-level data manipulation directly in Python.

What are Structured Arrays?

Standard NumPy arrays are homogeneous, meaning every element shares identical data types and sizes, such as 64-bit floats or 32-bit integers. In contrast, structured arrays allow each element to be treated as a composite record composed of multiple named sub-elements, or fields. Each field can hold a completely different data type, such as strings, integers, floating-point numbers, or nested sub-arrays.

This mechanism closely mirrors C struct types. Instead of maintaining multiple synchronized parallel arrays or relying on slower Python dictionaries, structured arrays store these multi-attribute records contiguously in memory.

Defining Custom Data Types (`dtype`)

To create a structured array, developers define a custom numpy.dtype specifying field names, data types, and optional shapes.

The most common approach uses a list of tuples, where each tuple specifies (field_name, data_type):

import numpy as np

# Define a C-like struct: { int32 id; char name[20]; float64 salary; }
employee_dtype = np.dtype([
    ('id', np.int32),
    ('name', 'U20'),
    ('salary', np.float64)
])

# Create the array with heterogeneous data
employees = np.array([
    (101, 'Alice', 75000.50),
    (102, 'Bob', 62000.00)
], dtype=employee_dtype)

In this example, every element in employees is a three-field record that occupies a fixed number of bytes in memory.

Memory Layout and C Compatibility

The primary reason structured arrays emulate C structures so effectively is their memory representation. Each element is arranged as a sequence of bytes corresponding to its defined fields:

Contiguous Allocation: The records are stored sequentially in a single memory buffer.
Field Offsets: Each field begins at a predefined byte offset from the start of the record.
Struct Padding (align=True): In C, compilers insert empty padding bytes between fields to align data on hardware word boundaries (e.g., 4-byte or 8-byte boundaries). By passing align=True to np.dtype, NumPy matches the exact byte layout and padding of a C compiler:

aligned_dtype = np.dtype([
    ('flag', np.int8),
    ('value', np.int64)
], align=True)

print(aligned_dtype.itemsize)  # Reflects the padded size, typically 16 bytes

Because of this binary compatibility, NumPy structured arrays can directly read, write, and map raw binary files, hardware buffers, or memory blocks shared with C and C++ extensions via libraries like ctypes or Cython.

Accessing and Manipulating Data

Structured arrays provide both column-oriented and row-oriented data access:

Field Access (Columns): Accessing a specific field name returns a view of that field across all records without copying memory:

salaries = employees['salary']  # Returns an array of float64
employees['salary'] *= 1.05     # Vectorized 5% raise applied in-place

Record Access (Rows): Indexing by position retrieves a single structured scalar, behaving similarly to a tuple with named attributes:
```
first_employee = employees[0]
print(first_employee['name'])  # Outputs: Alice
```

Key Advantages

Cache Efficiency: Keeping related heterogeneous attributes in contiguous blocks optimizes CPU cache utilization when iterating through complete records.
Vectorized Processing: Even with heterogeneous records, operations on individual fields remain vectorized, executing at compiled C speeds.
Low Overhead: Structured arrays consume substantially less memory than lists of Python objects or dictionaries by eliminating Python object header overhead for individual fields.