Python unicodedata: Using NFC and NFKD Normalization

This article provides an overview of how Python's built-in unicodedata module handles Unicode normalization, focusing on the NFC and NFKD forms. You will learn the difference between canonical and compatibility equivalence, understand the internal algorithmic process Python uses to decompose and compose characters, and see practical code examples to ensure consistent text processing.

Understanding Unicode Equivalence

In Unicode, visually identical characters can often be represented by different sequences of code points. For example, the character é can be represented as a single precomposed code point (\u00e9, Latin small letter e with acute) or as a decomposed sequence of two code points: the base letter e (\u0065) followed by the combining acute accent (\u0301).

Because these distinct byte sequences appear identical to users, comparing them directly in Python with the == operator will return False unless they are normalized. Normalization transforms text into a standardized binary representation based on two types of equivalence:

• Canonical Equivalence: Characters that have the same visual appearance and linguistic meaning (e.g., é as one code point vs. e + ´).
• Compatibility Equivalence: Characters that represent the same semantic construct but may differ in visual formatting or typography (e.g., the ligature fi, the circled digit ①, or superscript ²).

The Normalization Forms: NFC vs. NFKD

The unicodedata.normalize() function accepts four standard Unicode normalization forms: NFC, NFD, NFKC, and NFKD.

• NFD (Canonical Decomposition): Translates characters into their base components and separate combining marks using canonical equivalence.
• NFC (Canonical Decomposition, followed by Canonical Composition): First decomposes characters into canonical components, then immediately recombines them into single precomposed characters where possible. This is the most common form used for web content and general text storage.
• NFKD (Compatibility Decomposition): Decomposes characters using both canonical and compatibility mappings. It strips formatting distinctions—for instance, converting the superscript ² into a standard 2, or breaking the ligature fi into separate f and i characters.
• NFKC (Compatibility Decomposition, followed by Canonical Composition): Applies compatibility decomposition to reduce styled characters to standard equivalents, then applies canonical composition to merge compatible base characters and accents.

How unicodedata Executes Normalization

Python implements the unicodedata module through a compiled C extension (unicodedata.c) that packages tables derived from the official Unicode Character Database (UCD). When you invoke unicodedata.normalize(form, unistr), the module executes a multi-step algorithmic pipeline:

1. Decomposition Lookup: The engine iterates through the input string. For NFC, it queries the database for canonical decomposition mappings. For NFKD, it queries both canonical and compatibility decomposition mappings. If a character maps to multiple decomposed code points, the lookup recurses until no further decomposition is possible.
2. Canonical Ordering: Once decomposed, combining marks are sorted deterministically based on their Canonical Combining Class (CCC), an integer assigned by the Unicode standard. Marks with a CCC of zero (typically base characters) remain in place, while non-zero combining marks are sorted in ascending order by their CCC value.
3. Composition (NFC only): For NFC, the engine reads the ordered stream and attempts to combine the primary base character with subsequent combining marks. It references a two-character composition lookup table. A pair is merged into a single code point only if the character is not blocked by an intervening character with the same or higher combining class. Because NFKD is a decomposition-only format, this step is skipped for NFKD.

Practical Python Implementation

Normalization is executed using the unicodedata.normalize() method by passing the desired form as a string along with the target text.

import unicodedata

# Two visually identical strings with different code points
s1 = "\u00e9"  # Precomposed 'é'
s2 = "e\u0301"  # Decomposed 'e' + combining accent

print(s1 == s2)  # Output: False

# Normalizing to NFC (Canonical Composition)
nfc_s1 = unicodedata.normalize("NFC", s1)
nfc_s2 = unicodedata.normalize("NFC", s2)

print(nfc_s1 == nfc_s2)  # Output: True
print(len(nfc_s2))  # Output: 1

# Normalizing to NFKD (Compatibility Decomposition)
styled_text = "2⁵ and file"
nfkd_text = unicodedata.normalize("NFKD", styled_text)

print(nfkd_text)  # Output: "25 and file"

Use NFC when you need predictable equality checks without altering text layout or losing typographical distinction. Use NFKD when building search indexes, standardizing user input, or stripping accents and special formatting for downstream processing.