Subclassing Built-in Types in CPython

Subclassing built-in types like list or dict directly in CPython introduces subtle bugs and unexpected behaviors because CPython's underlying C implementations bypass overridden Python methods. This article explains why direct subclassing fails to trigger custom logic during internal operations and details the recommended alternatives, such as the collections module, to build robust custom data structures.

The Mechanism: C-Level Method Dispatch

CPython implements built-in data types in optimized C code. When a built-in method calls another method on the same object, it rarely uses the standard Python dynamic dispatch (Method Resolution Order). Instead, it calls fast C-level function pointers defined in the type's underlying C structs (such as tp_as_mapping or tp_as_sequence).

Because these internal C routines do not look up attributes dynamically through the instance's method resolution order, overridden methods written in Python are simply ignored by other built-in methods.

Example: Broken Method Overrides in dict

If you subclass dict and override __setitem__ to enforce custom logic, that logic only executes when using direct square-bracket assignment:

class CustomDict(dict):
    def __setitem__(self, key, value):
        super().__setitem__(key, f"modified_{value}")

d = CustomDict()
d['a'] = 'test'          # Calls CustomDict.__setitem__ -> 'modified_test'
d.update({'b': 'test'})  # Calls C-level dict.update -> 'test' (bypasses override)

In this scenario, d['a'] reflects the modified value, but d['b'] does not. The built-in dict.update() implementation directly accesses the internal hash table in C rather than routing the insertion through the Python __setitem__ method. The same problem occurs during instantiation: passing initial values via CustomDict(b='test') also bypasses your custom __setitem__.

A similar problem occurs with list. Overriding __getitem__ will affect index-based access, but built-in methods like extend(), list slicing, or internal iteration in C extensions will bypass your implementation.

Memory and Structural Changes

Directly subclassing a built-in type also alters its memory layout:

• Introduction of __dict__: Built-in types do not have an instance __dict__ by default to conserve memory. Subclassing them automatically adds an instance dictionary to hold dynamic attributes, increasing memory overhead per instance unless explicit __slots__ = () is declared.
• Immutability Constraints: Subclassing immutable types like int, str, or tuple requires overriding __new__ instead of __init__, because the underlying object is already immutable by the time __init__ is called.

The Standard Solution: collections.UserDict and collections.UserList

To solve this problem, Python provides wrapper classes in the standard library: collections.UserDict, collections.UserList, and collections.UserString.

Unlike the built-ins, these classes are implemented in pure Python. They store an underlying built-in instance in a .data attribute and route all operations through standard Python methods:

from collections import UserDict

class ReliableDict(UserDict):
    def __setitem__(self, key, value):
        super().__setitem__(key, f"modified_{value}")

d = ReliableDict()
d['a'] = 'test'          # 'modified_test'
d.update({'b': 'test'})  # 'modified_test'

Because UserDict.update() is written in Python, it delegates its operations to __setitem__, ensuring your custom behavior is uniformly applied.

The Alternative: collections.abc

When you need total control over data storage rather than wrapping a standard container, inherit from the Abstract Base Classes in collections.abc (such as MutableMapping or MutableSequence). Implementing the required abstract methods ensures that your class behaves like a native container while providing full control over dispatch, validation, and storage.