Python String Interning for Short Strings

String interning is an optimization mechanism in CPython—the standard Python implementation—designed to conserve memory and accelerate comparison operations by storing only one copy of distinct, immutable string values. When Python encounters specific string literals, identifiers, or single-character strings, it places them in an internal lookup table. Subsequent references to identical strings point to the existing object in memory rather than allocating a new one. This guide explains how Python detects candidate strings, handles identifiers and short strings at compile time, and manages this internal caching process behind the scenes.

What Is String Interning?

In Python, strings are immutable. Because their values cannot change after creation, having multiple identical string objects spread across memory is often redundant. String interning resolves this by ensuring that identical interned strings share the exact same memory address.

This provides two primary benefits:

1. Reduced Memory Footprint: Repetitive strings occupy memory only once.
2. Faster Comparisons: Comparing two interned strings evaluates pointer equality (is) rather than a character-by-character check (==). Pointer comparison runs in \(O(1)\) constant time.

Automatic Interning of Identifiers

CPython automatically interns strings that resemble valid Python identifiers. These are strings consisting exclusively of ASCII letters, digits, and underscores (the pattern [a-zA-Z0-9_]*).

Because Python relies heavily on dictionaries for runtime operations—such as attribute lookups (object.__dict__), global variables (globals()), and local scopes—variable, function, class, and method names are repeatedly compared. Interning these identifier-like strings ensures that internal dictionary lookups execute as quickly as possible.

# Identifier-like strings are automatically interned
a = "python_identifier"
b = "python_identifier"
print(a is b)  # True

# Strings with special characters or spaces typically are not
c = "not an identifier!"
d = "not an identifier!"
print(c is d)  # May evaluate to False depending on the context

Interning Short Strings and Constants

CPython optimizes small or frequently used strings through specific internal rules:

• 0 and 1-Character Strings: All single-character ASCII strings (length 0 or 1) are interned or statically cached by CPython during interpreter initialization.
• Compile-Time Constants: During the compilation of Python source code to bytecode, the compiler's code-optimization stage (formerly the peephole optimizer, now integrated into the AST compiler) automatically interns constant string literals up to a specific length limit (historically 4096 characters in CPython), provided they meet identifier criteria.

Strings generated dynamically at runtime (for example, through concatenation or user input) generally bypass automatic interning, even if they look like identifiers:

s1 = "hello"
s2 = "".join(["h", "e", "l", "l", "o"])

print(s1 == s2)  # True (same value)
print(s1 is s2)  # False (different objects in memory)

Under the Hood: The CPython Implementation

At the C level, CPython tracks interned strings using an internal dictionary named interned, which maps string objects to themselves.

When the interpreter processes an internable string:

1. It calls the internal C-API function PyUnicode_InternInPlace(PyObject **p).
2. The function checks the global interned dictionary.
3. If the string already exists in the dictionary, *p is adjusted to point to the cached string, and the reference count of the duplicate is decremented.
4. If the string does not exist, it is added to the dictionary.

Starting in Python 3.12, CPython introduced immortal objects for core built-ins and static strings. Immortal strings remain in memory permanently without reference counting overhead, eliminating modifications to the reference count during lookups.

Manual Interning with sys.intern()

Developers can manually intern dynamically generated strings using the sys.intern() function. This is particularly useful when building large data structures that store recurring string values, such as natural language processing vocabularies or database record parsers.

import sys

# Manually interning dynamic strings
s1 = sys.intern("".join(["dynamic", "_", "token"]))
s2 = sys.intern("dynamic_token")

print(s1 is s2)  # True

Once passed through sys.intern(), the string reference is registered into CPython's intern table, allowing subsequent lookups and comparisons to achieve maximum memory efficiency and \(O(1)\) equality checks.