WordPiece vs. BPE: Subword Merging in Artificial Intelligence

This article explores the fundamental differences between WordPiece and Byte Pair Encoding (BPE) tokenization methods in artificial intelligence. Specifically, it focuses on how WordPiece selects subword merges by maximizing the likelihood of the training corpus, contrasting this probabilistic approach with BPE's purely frequency-based merging strategy.

Tokenization is a critical preprocessing step in natural language processing (NLP) that breaks down text into smaller units called tokens. Both BPE and WordPiece are subword tokenization algorithms designed to handle out-of-vocabulary words by splitting rare words into meaningful subword units. Both algorithms start with a base vocabulary of individual characters and iteratively merge them to create new subword tokens until a predefined vocabulary size is reached. However, the exact mathematical criteria they use to decide which characters or subwords to merge at each step is what sets them apart.

Byte Pair Encoding takes a straightforward, greedy approach based on raw frequency. During training, BPE scans the corpus and counts how often adjacent pairs of tokens appear together. It then merges the single most frequent pair into a new token and adds it to the vocabulary. For example, if the pair "t" and "h" appears more often than any other pair, BPE merges them into "th". It does not consider how often the individual characters "t" and "h" appear on their own; it only cares about the total count of the pair.

WordPiece, famously used in models like BERT, modifies this merging process by using a probabilistic approach. Instead of selecting the most frequent pair, WordPiece selects the merge that maximizes the likelihood of the training data when evaluated by an unigram language model. It calculates a score for each potential pair by dividing the frequency of the pair by the product of the frequencies of its individual parts. The formula is:

Score = (Frequency of Pair) / (Frequency of First Element × Frequency of Second Element)

By using this formula, WordPiece prioritizes merging pairs where the individual elements rarely appear apart from one another. Looking back at the previous example, the letters "t" and "h" are extremely common in the English language and appear in many different combinations. Because their individual frequencies are very high, the denominator in the WordPiece formula becomes massive, resulting in a lower merge score for "th".

Conversely, consider a pair of subwords like "un" and "believable". If these two tokens appear together frequently relative to how often they appear separately in the specific training corpus, their merge score will be high. WordPiece essentially asks whether the two tokens occur together more often than would be expected by random chance.

By maximizing the likelihood of the training corpus at every step, WordPiece ensures that the vocabulary is populated with subwords that carry strong mutual information. This prevents the algorithm from aggressively merging extremely common characters early on just because they are common. Consequently, WordPiece builds a more statistically optimized vocabulary for language modeling tasks compared to the raw frequency counts of Byte Pair Encoding.