Unicode Normalization in Lodash String Methods

This article examines how Unicode normalization disparities affect Lodash string functions, detailing how decomposed characters, diacritics, and combining marks lead to unexpected behavior in operations like truncation, deburring, and pattern matching. While Lodash provides broad utility for text transformation, its lack of built-in Unicode normalization can result in silent data corruption, failed string comparisons, and broken character rendering.

Composed vs. Decomposed Unicode Forms

Unicode allows certain characters to be represented in multiple equivalent ways:

Visually, both forms render identically in modern browsers and terminals. However, at the byte and code point level, their internal structures differ significantly.

Where Lodash Fails with Unnormalized Strings

Lodash includes custom regular expressions to handle astral symbols, emoji sequences, and surrogate pairs better than legacy ECMAScript 5 implementations. Despite these enhancements, Lodash does not normalize strings to a consistent Unicode form (NFC or NFD) before processing. This leads to distinct failure modes across several functions.

1. Inconsistent Searching and Comparison (_.includes, _.startsWith, _.endsWith)

Lodash’s search utilities rely on native character code evaluation. If the target string uses decomposed Unicode (NFD) and the search term uses precomposed Unicode (NFC), the operation will return false despite visual equivalence.

const nfcString = 'café'; // 'cafe\u00E9'
const nfdString = 'cafe\u0301'; // 'cafe' + combining acute accent

_.includes(nfcString, 'é'); // true
_.includes(nfdString, 'é'); // false (searches for \u00E9, but string contains \u0301)

2. Broken Diacritics in _.deburr

The _.deburr method converts Latin-1 Supplement and Latin Extended-A characters to basic Latin letters by mapping individual precomposed code points.

When passed an NFD string, _.deburr processes the base ASCII character, leaves it untouched, and subsequently fails to strip the trailing combining mark. This results in the mark detaching and attaching to the following character, or remaining as a standalone orphan mark:

const composed = '\u00E9'; // 'é' (NFC)
const decomposed = 'e\u0301'; // 'e' + combining accent (NFD)

_.deburr(composed);   // 'e' (correct)
_.deburr(decomposed); // 'e\u0301' (accent remains intact)

3. Splitting and Truncation Hazards (_.truncate, _.slice)

Lodash tries to avoid splitting surrogate pairs when computing lengths or truncating strings. However, combining characters in NFD strings are distinct code points from their base character.

When _.truncate measures character counts or cuts a string to a specific limit, it can cut between the base character and the combining mark:

4. Word Tokenization Bugs in _.words, _.camelCase, and _.kebabCase

Functions that parse strings into words rely on regular expressions that delineate word boundaries. Combining diacritical marks outside the standard NFC block are often classified as non-word characters or unexpected symbols.

This causes Lodash to split a single semantic character across word boundaries during case transformations:

// NFD representation of 'münchen'
const cityNFD = 'mu\u0308nchen'; 

_.words(cityNFD); 
// May tokenize into ['mu', 'nchen'] because \u0308 is treated as a delimiter

Remediation: Pre-Normalization

Lodash does not provide a normalization flag in its string methods. To prevent normalization bugs, you must normalize strings using JavaScript’s native String.prototype.normalize() before passing them to Lodash:

function safeDeburr(str) {
  // Normalize to NFC or NFD depending on requirements
  return _.deburr(str.normalize('NFC'));
}

function safeIncludes(collection, target) {
  return _.includes(collection.normalize('NFC'), target.normalize('NFC'));
}