Python Unicode Storage: ASCII vs UCS-1, UCS-2, UCS-4

Python optimizes in-memory string management through the Flexible String Representation introduced in PEP 393. This article examines the architectural differences between compact ASCII strings and UCS-1, UCS-2, and UCS-4 Unicode representations in CPython, explaining how character ranges, memory footprints, and underlying C structures distinguish each format.

The Flexible String Representation (PEP 393)

Prior to Python 3.3, CPython relied on either a "narrow" build using 2-byte units (UCS-2) or a "wide" build using 4-byte units (UCS-4). This approach either wasted vast amounts of memory for basic Latin text or failed to support characters outside the Basic Multilingual Plane (BMP) without complex surrogate pairs.

PEP 393 solved this by selecting the minimal required byte-width for a string based on the maximum Unicode code point it contains. All strings are stored as fixed-width arrays, preserving \(O(1)\) random character access by index.

Compact ASCII Strings

A string is classified as compact ASCII when every character falls strictly within the 7-bit ASCII range (\(U+0000\) to \(U+007F\)).

• Character Width: 1 byte per character.
• Underlying C Structure: Stored using PyASCIIObject.
• Zero-Copy UTF-8 Sharing: Because pure ASCII bytes are identical in native representation and UTF-8 encoding, the UTF-8 pointer in the C structure points directly to the ASCII character buffer. This eliminates the need to allocate separate memory when exporting the string to UTF-8.
• Header Size: Has the smallest header overhead among Python string types (48 bytes on 64-bit systems).

UCS-1 (Latin-1) Representation

A string uses the UCS-1 representation when its maximum code point is between \(U+0080\) and \(U+00FF\).

• Character Width: 1 byte per character.
• Underlying C Structure: Stored using PyCompactUnicodeObject.
• UTF-8 Discrepancy: Unlike ASCII, code points in the Latin-1 supplement (\(128\) to \(255\)) require 2 bytes when encoded in UTF-8. Consequently, Python cannot share the internal buffer with its UTF-8 representation; UTF-8 serialization requires additional memory allocation and caching.
• Header Size: Requires a larger header (72 bytes on 64-bit systems) to track additional encoding states and potential UTF-8 cached pointers.

UCS-2 Representation

Python escalates a string to UCS-2 when at least one code point exceeds \(U+00FF\), but no code points exceed \(U+FFFF\).

• Character Width: 2 bytes per character.
• Scope: Covers the entire Basic Multilingual Plane (BMP), including Cyrillic, Greek, Hebrew, Arabic, and most standard CJK (Chinese, Japanese, Korean) ideographs.
• Memory Impact: Consumes twice the buffer size of an ASCII or UCS-1 string of equal length. Even if only a single character requires 2 bytes, every character in the string is allocated 2 bytes to maintain \(O(1)\) indexing.

UCS-4 Representation

When a string contains even a single code point greater than \(U+FFFF\), Python allocates it using UCS-4.

• Character Width: 4 bytes per character.
• Scope: Accommodates code points up to \(U+10FFFF\), which includes emojis, mathematical alphanumerics, historical scripts, and uncommon CJK extension ideographs.
• Memory Impact: Represents the heaviest storage format, quadrupling the raw character buffer size compared to ASCII.

Key Distinctions Summary