How Lodash _.words Uses Regex to Extract Words

The Lodash _.words method extracts words from a string by applying regular expressions, either through a user-provided custom pattern or through its own built-in regular expressions designed to handle both standard ASCII and complex Unicode text. By evaluating whether a string contains Unicode sequences or standard characters, the method selects an appropriate regular expression to correctly capture alphanumeric tokens, compound words, contractions, and symbols while discarding unwanted whitespace and punctuation.

The Custom Pattern Approach

When a developer passes a custom regular expression as the second argument to _.words(string, [pattern]), Lodash bypasses its internal parsing logic. It directly invokes JavaScript's native String.prototype.match() method:

return pattern ? (string.match(pattern) || []) : /* internal logic */;

If the custom pattern matches substrings, an array of those matches is returned; otherwise, it returns an empty array.

Default Detection: ASCII vs. Unicode

When no pattern is provided, Lodash determines whether the input string requires Unicode-aware parsing or if standard ASCII processing is sufficient. It uses an internal helper function, typically named hasUnicodeWord, which tests the string against a regex that looks for non-ASCII characters, zero-width joiners, surrogates, or combining marks.

Based on this check, Lodash routes the string to one of two specialized regex routines: asciiWords or unicodeWords.

ASCII Word Extraction

If the string contains only standard ASCII characters, Lodash employs a streamlined regex pattern designed to capture sequences of characters while ignoring standard punctuation and control characters.

The typical regular expression used for ASCII strings is:

/[^\x00-\x2f\x3a-\x40\x5b-\x60\x7b-\x7f]+/g

This pattern uses a negated character class that excludes:

• ASCII control characters (\x00-\x1f and \x7f)
• Common punctuation and symbols such as spaces, exclamation marks, and slashes (\x20-\x2f)
• Colons, semicolons, and comparison operators (\x3a-\x40)
• Brackets, backslashes, and carets (\x5b-\x60)
• Braces and tildes (\x7b-\x7e)

Any contiguous sequence of characters outside these excluded ranges is matched as a single word, effectively capturing words separated by spaces, commas, hyphens, and other punctuation.

Unicode Word Extraction

When strings contain characters outside standard ASCII—such as accented letters, scripts without whitespace separation, contractions, or emojis—the ASCII pattern fails to segment text accurately. In these cases, Lodash constructs a comprehensive, multi-part regular expression (reUnicodeWord).

The Unicode regular expression accounts for:

• Letter casing and transitions: Sequences of uppercase letters followed by lowercase letters (handling camelCase and PascalCase splitting).
• Ordinals and numbers: Digits grouped with standard numbering conventions.
• Contractions and possessives: Letters joined by apostrophes or typographical quotation marks (such as 's or 't).
• Diacritics and combining marks: Base characters combined with accent marks or modifiers.
• Emojis and surrogate pairs: Multi-byte symbols, skin tone modifiers, and zero-width joiner sequences (\u200d).

By matching these specific linguistic and structural rules instead of merely excluding delimiters, the Unicode regex isolates discrete semantic tokens without fragmenting multi-byte characters or emoji sequences.

Summary of the Extraction Flow

1. Check for Custom Pattern: If a regex is explicitly passed, execute string.match(pattern) and return the result.
2. Inspect Encoding: If no pattern is provided, evaluate the string with hasUnicodeWord.
3. Execute Internal Regex: Apply the unicodeWords regex if Unicode characteristics are detected; otherwise, apply the asciiWords regex.
4. Return Results: Return the matched tokens as an array, or an empty array if no matches exist.