How Tokenization Splits Text into Subword Units for LLMs

This article provides a comprehensive overview of how tokenization transforms raw human text into subword units for Large Language Models (LLMs). Because artificial intelligence systems cannot process raw characters directly, they rely on tokenization to break sentences down into manageable pieces before converting them into numerical data. We will explore the limitations of older word-based and character-based methods, the mechanics of modern subword tokenization, and the specific algorithms—such as Byte-Pair Encoding (BPE), WordPiece, and Unigram—that power today's leading AI systems.

The Need for Subword Tokenization

Large Language Models only understand numbers. To process text, an LLM must first use a tokenizer to split raw strings into smaller pieces, called tokens, and map those tokens to numerical IDs in a fixed vocabulary.

Historically, AI systems used either word-level or character-level tokenization, both of which have major flaws:

  • Word-level tokenization splits text strictly by spaces and punctuation. This requires a massive vocabulary to account for every word variation (e.g., "run", "runs", "running"). If the model encounters a new or misspelled word, it flags it as an "unknown" token, losing all meaning.
  • Character-level tokenization splits text into individual letters. While this requires a very small vocabulary and eliminates unknown words, it destroys the inherent meaning of word groupings. Furthermore, a simple sentence becomes a massive sequence of tokens, requiring excessive computational power to process.

Subword tokenization solves both problems. It operates on the principle that frequently used words should remain whole, while rare words should be split into smaller, meaningful chunks (prefixes, roots, and suffixes).

How the Splitting Process Works

When you feed raw text into an LLM, the subword tokenizer processes it through a straightforward pipeline:

  1. Normalization: The raw text is cleaned. Depending on the model, this might involve removing unnecessary whitespace, stripping accents, or converting letters to lowercase.
  2. Pre-tokenization: The text is broadly split into basic words using spaces and punctuation as boundaries. For example, the phrase "unhappily running" becomes ["unhappily", "running"].
  3. Subword Splitting: The tokenizer applies a trained algorithm to break complex or rare words into smaller units present in its vocabulary. "Unhappily" might be split into ["un", "happi", "ly"], and "running" into ["run", "ning"]. Common words like "the" or "and" remain intact as single tokens.
  4. Mapping to IDs: Each subword token is matched to its corresponding unique integer in the model's vocabulary. These numbers are then fed into the neural network.

By splitting "unhappily" into three distinct pieces, the LLM can use its understanding of the prefix "un" (not), the root "happi" (happy), and the suffix "ly" (adverb) to deduce the meaning of the word, even if it has rarely seen the exact combination before.

Core Subword Algorithms

Different LLMs use different algorithms to decide how to build their vocabularies and split raw text. The three most common methods are Byte-Pair Encoding, WordPiece, and Unigram.

Byte-Pair Encoding (BPE)

Used by models like OpenAI's GPT series and Meta's Llama, BPE is a bottom-up algorithm. During training, it starts with a vocabulary of individual characters. It scans the training text, finds the two characters that appear next to each other most frequently (e.g., "t" and "h"), and merges them into a new subword token ("th"). It repeats this process thousands of times, merging characters into larger and larger subwords until it reaches a pre-set vocabulary size. Modern LLMs use Byte-level BPE, which applies this process to raw UTF-8 bytes rather than characters, ensuring the model can process any language, symbol, or emoji without ever encountering an unknown token.

WordPiece

Developed by Google for models like BERT, WordPiece is very similar to BPE. It also starts with individual characters and merges them iteratively. However, instead of merging pairs based strictly on raw frequency, WordPiece uses a likelihood score. It merges two tokens only if the combination appears frequently relative to the frequency of the individual tokens appearing separately. This ensures that the algorithm prioritizes merges that carry unique linguistic value. WordPiece typically marks subwords that do not start a word with a double hashtag, such as ["run", "##ning"].

Unigram

Often used in conjunction with Google's SentencePiece framework (found in models like T5 and ALBERT), the Unigram algorithm works in reverse. Instead of starting small and merging up, it takes a top-down approach. It begins with a massive vocabulary containing all possible words and subword combinations from the training data. It then iteratively removes tokens that contribute the least to the overall probability of representing the training text. It whittles the list down until only the most efficient subword units remain, reaching the desired vocabulary size. When splitting new text, Unigram calculates the mathematically most probable way to divide a word based on the surviving tokens.