How Lodash Words Handles Emojis and Graphemes

The Lodash _.words function extracts individual words from a string, relying on specialized Unicode-aware regular expressions to correctly parse complex characters, emojis, and grapheme clusters. Instead of naively splitting strings by whitespace or treating characters as 16-bit code units, Lodash detects Unicode sequences to keep multi-codepoint emojis, zero-width joiners, skin-tone modifiers, and diacritics intact as distinct tokens or parts of words.

Unicode Detection Mechanism

When invoked without a custom pattern argument, _.words determines whether to use a standard ASCII regular expression or an advanced Unicode regular expression. It first evaluates the input string using an internal utility called hasUnicodeWord.

If the string contains only basic Latin and standard ASCII characters, Lodash employs a simpler, high-performance regular expression (asciiWords) to isolate alphanumeric sequences. If the string contains extended Unicode characters, mathematical symbols, or emoji codepoints, _.words switches to unicodeWords, which accommodates multibyte sequences and compound graphemes.

Parsing Emojis and Modifiers

In standard JavaScript, emojis often consist of surrogate pairs, where a single visible character comprises two 16-bit code units. Complex emojis can span several codepoints joined by special invisible control characters. Lodash's unicodeWords pattern explicitly accounts for these structures:

Surrogate Pairs: High and low surrogates ([\uD800-\uDBFF][\uDC00-\uDFFF]) are matched together, preventing surrogate pairs from being split into unprintable or corrupted fragments.
Skin-Tone Modifiers: Emojis using the Fitzpatrick scale (codepoints \U+1F3FB through \U+1F3FF) are bound to their preceding base emoji character. When _.words processes an emoji sequence such as 👍🏽, it evaluates the base symbol and the tone modifier as a single, contiguous word token.
Zero-Width Joiner (ZWJ) Sequences: Complex compound emojis—such as the technologist (👨‍💻) or diverse family combinations—use Zero-Width Joiners (\u200D) to connect multiple base emojis. Lodash's internal regular expressions capture the base emojis, the joiners, and any trailing variation selectors (\uFE0F) as a unified element.
Flags and Regional Indicators: Flag emojis are made up of pairs of Regional Indicator Symbols (codepoints \U+1F1E6 through \U+1F1FF). Lodash matches these paired symbols together, ensuring that a flag like 🇺🇸 is treated as one complete word token rather than two separate alphabetic components.

Complex Graphemes and Diacritics

Beyond emojis, human languages often employ complex grapheme clusters where a single perceived character consists of a base letter accompanied by one or more combining diacritical marks (such as accents, umlauts, or tildes).

In strings where characters are written in decomposed form (such as e followed by the combining acute accent \u0301 instead of the precomposed é), naive string splits would separate the accent from the base character. Lodash’s unicodeWords includes patterns for combining marks ([\u0300-\u036F\u1DC0-\u1DFF...]), ensuring that modifying diacritics remain attached to their base letter and do not generate isolated punctuation-like tokens.

Tokenization Behavior and Custom Patterns

When emojis appear in a sentence alongside text, _.words treats standalone emojis as standalone words:

_.words('Build 🚀 fast apps');
// Output: ['Build', '🚀', 'fast', 'apps']

When an emoji is joined directly to text without spaces, the regex engine evaluates the boundary between the alphanumeric character class and the emoji character class, separating them into distinct tokens:

_.words('Launch🚀Now');
// Output: ['Launch', '🚀', 'Now']

Passing a custom regular expression to _.words(string, [pattern]) completely overrides this default Unicode handling. If a custom pattern does not explicitly account for surrogate pairs, variation selectors, and ZWJ sequences, complex graphemes and emojis may be split into raw surrogate components or discarded altogether.