Understanding String Normalization in JavaScript

String normalization is the process of converting text into a standard, unique binary representation to ensure accurate string comparisons, searching, and sorting across different Unicode representations. In JavaScript, strings that appear visually identical can have different underlying byte sequences due to combining characters. The built-in String.prototype.normalize() method resolves this problem by converting characters into a specified Unicode Normalization Form, guaranteeing that equivalent characters evaluate as equal.

Why String Normalization is Necessary

Unicode allows certain characters to be represented in multiple ways. A common example is an accented character such as é:

  1. Precomposed character: \u00E9 (a single code point representing é)
  2. Decomposed sequence: \u0065\u0301 (e followed by the combining acute accent \u0301)

While both render identically on screen as é, a strict comparison (===) in JavaScript will evaluate to false because their code units do not match:

const str1 = '\u00E9';       // "é"
const str2 = '\u0065\u0301'; // "é"

console.log(str1 === str2); // false
console.log(str1.length);   // 1
console.log(str2.length);   // 2

String normalization transforms these variations into a uniform format so they can be reliably compared, stored, or searched.

How String.prototype.normalize() Works

The normalize() method returns the Unicode Normalization Form of the calling string. It accepts an optional string parameter representing the normalization form to use.

Syntax

str.normalize([form]);

If no argument is passed, it defaults to "NFC".

The Four Normalization Forms

Unicode defines four normalization forms, categorized by canonical vs. compatibility equivalence:

  1. NFC (Canonical Decomposition, followed by Canonical Composition)
    • Decomposes characters and recomposes them into precomposed characters where available.
    • Best for standard web text and general comparisons.
    const str1 = '\u0065\u0301'; // "e" + accent
console.log(str1.normalize('NFC') === '\u00E9'); // true
  2. NFD (Canonical Decomposition)
    • Decomposes combined characters into their base characters and separate combining marks.
    • Useful when you need to strip accents or analyze base characters.
    const str = 'é';
console.log(str.normalize('NFD')); // "e\u0301"
  3. NFKC (Compatibility Decomposition, followed by Canonical Composition)
    • Replaces formatting variants (such as ligatures, superscripts, or full-width characters) with their standard equivalents and recomposes them canonically.
    const ligature = 'fi'; // Single ligature character "fi"
console.log(ligature.normalize('NFKC')); // "fi"
  4. NFKD (Compatibility Decomposition)
    • Decomposes formatting variants into their basic constituent parts without recomposing them.
    const fraction = '½';
console.log(fraction.normalize('NFKD')); // "1/2"

Common Use Cases

1. Reliable String Comparison

Normalize user inputs and database records to avoid false negatives when querying or verifying input:

function areStringsEqual(a, b) {
  return a.normalize('NFC') === b.normalize('NFC');
}

console.log(areStringsEqual('\u00E9', '\u0065\u0301')); // true

2. Removing Accents and Diacritics

You can strip diacritics from text for search indexing by decomposing characters with NFD and removing the combining marks using a regular expression:

function removeAccents(str) {
  return str.normalize('NFD').replace(/[\u0300-\u036f]/g, '');
}

console.log(removeAccents('Crème brûlée')); // "Creme brulee"

Use NFKC to ensure users searching with standard ASCII characters can match formatted Unicode characters:

const query = '2';
const databaseEntry = '²'; // Superscript two

console.log(query === databaseEntry.normalize('NFKC')); // true