Source: Google Research
Introduction
In modern Natural Language Processing (NLP), subword tokenization has become a key strategy for bridging the gap between word-level and character-level representations. Splitting text strictly by whitespace can cause issues with unseen words, yielding an abundance of <unk> tokens. Conversely, a purely character-based scheme can explode sequence lengths and make it harder for models to capture higher-level word relationships.
In a previous article, we explored Byte Pair Encoding (BPE) as one popular subword technique. Here, we turn our attention to WordPiece – another highly influential subword method that gained prominence through Google’s BERT and related Transformer models. WordPiece helps reduce unknown words while striking a balance between vocabulary size and linguistic flexibility.
- Explain the theoretical underpinnings of WordPiece and how it emerged
- Walk through the core algorithm and an example (“uncommon”)
- Demonstrate real-world use, including code snippets, domain considerations, and best practices
- Discuss common pitfalls and potential alternatives to WordPiece
By the end, you’ll see how WordPiece tokenization works, why it’s so widespread in Transformer-based models, and how you might apply or adapt it for your own NLP tasks.
Visual Overview: Understanding Tokenization Methods
Before diving deep into WordPiece, let’s visualize how different tokenization methods work:
Source: ByteByteGo
Historical Perspective & Core Principles
Origins
WordPiece traces its roots to work by Schuster and Nakajima (2012) at Google, initially focused on improving voice search for Japanese and Korean. Over time, it proved robust for machine translation and other text-heavy tasks. The breakthrough that propelled WordPiece into the mainstream, however, came when Devlin et al. (2018) adopted it in BERT (Bidirectional Encoder Representations from Transformers). Because BERT achieved state-of-the-art results across many NLP benchmarks, WordPiece became a de facto standard in tokenization pipelines.
Conceptually, WordPiece is similar to BPE in that it segments words into subwords (e.g., turning “uncommon” into [“un”, “##common”]). However, the vocabulary-building process differs in a crucial way: instead of simply merging the most frequent pairs like BPE, WordPiece uses a likelihood-based scoring mechanism that measures how each potential subword addition impacts the overall probability of the training corpus. This is distinct from BPE’s more direct frequency-based merges and gives WordPiece a slightly different behavior, especially on large corpora.
Key Innovations
- Likelihood-Based Scoring: Unlike BPE’s pure frequency approach, WordPiece computes a score for each candidate pair. The algorithm selects pairs that maximize the likelihood of the training data when merged. The scoring formula is: score(a, b) = freq(ab) / (freq(a) × freq(b)). This prioritizes merging pairs where the individual parts are less frequent in the vocabulary relative to their joint occurrence.
- Longest-Match-First Strategy: WordPiece uses a greedy longest-match-first approach (also known as maximum matching or MaxMatch) to find the longest possible subword that exists in the vocabulary, then recursively processes the remaining part
- Special Markers: WordPiece implementations use “##” (in BERT) to denote subword continuation, which are key for reconstructing words during decoding
Why It Matters
By using a likelihood-driven approach to build subword units, WordPiece excels at reducing unknown tokens and handling morphological nuances. The scoring mechanism measures the association strength between tokens-pairs that occur more often than expected given their individual frequencies get priority. This tends to produce more linguistically meaningful subwords compared to BPE’s simple frequency approach. It’s particularly effective in Transformer-based architectures and has become the standard tokenization method for models like BERT, DistilBERT, Electra, and their variants.
Inside WordPiece: Algorithm & Implementation
Vocabulary Construction
WordPiece starts with a minimal set of characters much like BPE might. The difference lies in how it selects new tokens to add:
Initial Set: A vocabulary of single characters (or bytes) covers any possible text, plus special tokens like [UNK], [CLS], [SEP], [MASK], and [PAD].
Scoring Subwords: Each candidate subword pair is evaluated using the likelihood-based scoring formula. The algorithm identifies which merge would most increase the likelihood of the training data.
Iterative Expansion: The tokenizer picks the best-scoring subword merge one at a time and adds it to the vocabulary. This process repeats until a desired vocabulary size is reached (e.g., 30k or 50k tokens) or until the likelihood improvement falls below a threshold.
The Scoring Mechanism Explained
The key difference from BPE lies in the scoring mechanism. WordPiece doesn’t just count frequencies, it evaluates how strongly two tokens co-occur relative to their individual frequencies.
Given two tokens a and b, the score is computed as: score(a, b) = freq(ab) / (freq(a) × freq(b)). This can also be expressed probabilistically as: score(a, b) = P(ab) / (P(a) × P(b)) = P(b|a) / P(b), where P(ab) is the probability of the pair appearing together, and P(a) and P(b) are the individual probabilities of each token.
This scoring formula measures the conditional probability of b given a, divided by the marginal probability of b. If the score is high (greater than 1), it means the tokens occur together more often than would be expected by chance, making them good candidates for merging. This approach tends to favor merges that form more linguistically meaningful units rather than just the most frequent pairs.
Key Difference from BPE: Unlike BPE which merges the most frequent pairs, WordPiece uses this likelihood-based scoring mechanism that considers the impact on overall corpus probability. For instance, it won’t necessarily merge (“un”, “##able”) even if that pair occurs very frequently, because both “un” and “##able” likely appear in many other words and have high individual frequencies. This allows WordPiece to make more informed decisions about which merges will be most beneficial.
Prefix and Word Boundaries
An important hallmark of WordPiece is the prefix, commonly seen in models like BERT. For English or similar languages, a word that’s split mid-way might produce something like: [“un”, “##common”]. The “##” indicates that the token should be concatenated to the previous subword (minus the space) during decoding, ensuring you get “uncommon” as a single continuous word.
This marker system is crucial for proper reconstruction during decoding and helps maintain word boundaries while still allowing for subword segmentation. It distinguishes word-initial tokens (without ##) from continuation tokens (with ##).
Training Algorithm Summary
The WordPiece training process can be summarized as follows:
- Initialize the vocabulary with all base characters and special tokens
- Build a language model on the training data using the current vocabulary
- Score all possible pairs of adjacent tokens using the likelihood formula
- Select the best-scoring pair and add it to the vocabulary
- Update the corpus tokenization with the new merged token
- Repeat until reaching the target vocabulary size or until likelihood improvements become negligible
Important Note: Google never open-sourced the exact training implementation of WordPiece, so implementations in libraries like Hugging Face are based on published literature and may not be 100% identical to the original.
Comparing WordPiece with BPE and SentencePiece
Tokenization Workflow
- BPE: Iteratively merges the most frequent pairs of tokens (initially characters). Simple frequency-based approach.
- WordPiece: Uses likelihood-based scoring to select merges that maximize corpus probability. More sophisticated than raw frequency.
- SentencePiece (Unigram): Starts with a large subword set and prunes subwords that don’t contribute significantly to corpus likelihood. Top-down approach versus bottom-up.
Scoring Mechanisms
BPE uses straightforward frequency counting for merge, it simply picks the most frequent pair at each step. This is simple and fast but can sometimes merge pairs that aren’t linguistically meaningful.
WordPiece uses a more sophisticated scoring mechanism that considers both frequency and the impact on vocabulary coverage. The formula score(a, b) = freq(ab) / (freq(a) × freq(b)) prioritizes pairs with strong association, making it potentially more robust for diverse text types and leading to more linguistically meaningful subwords.
SentencePiece with Unigram uses expectation-maximization, starting with many tokens and pruning the least useful. This top-down approach offers more flexibility, especially for languages without clear word boundaries.
Special Markers and Tokenization
WordPiece uses “##” markers to indicate subword continuation, which is crucial for proper reconstruction during decoding. This marker system helps maintain word boundaries while still allowing for subword segmentation, making it particularly effective for languages with clear word boundaries like English.
Key difference: WordPiece only saves the final vocabulary, not the merge rules learned. During tokenization, it uses longest-match-first to find subwords in the vocabulary. BPE, on the other hand, saves and applies merge rules sequentially.
Real-World Usage
- WordPiece: BERT, DistilBERT, Electra, MobileBERT, MPNET
- BPE: GPT-2, GPT-3, RoBERTa, BART
- SentencePiece: T5, ALBERT, XLNet, mBART
- Many open-source frameworks (Hugging Face, Fairseq, TensorFlow Text) provide easy WordPiece integrations, reflecting its widespread adoption
Hands-On with WordPiece
Using Established Libraries
While you can write a custom WordPiece implementation, most practitioners use off-the-shelf libraries, particularly in conjunction with popular Transformer models. For instance, Hugging Face’s transformers library provides pretrained tokenizers for BERT, RoBERTa, and more. Under the hood, many of these use WordPiece or a very similar subword scheme.
from transformers import AutoTokenizer
# Load a pretrained BERT tokenizer (WordPiece under the hood)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
text = "WordPiece is awesome!"
encoded = tokenizer.encode_plus(
text,
add_special_tokens=True, # e.g., [CLS], [SEP]
return_tensors='pt'
)
print("Token IDs:", encoded["input_ids"])
print("Tokens:", tokenizer.convert_ids_to_tokens(encoded["input_ids"][0]))The “encode_plus” method processes the text into WordPiece tokens, adding special markers like [CLS] (classification token at the start) and [SEP] (separator token at the end). The resulting IDs can be mapped back to textual subwords with convert_ids_to_tokens, often revealing “word” + “##piece” or “aw” + “##esome.”
A Worked Example: “uncommon”
To see how WordPiece might handle a single word, consider “uncommon.”
Initial Tokenization: A naive word-level approach yields just [“uncommon”], which might become <unk> if “uncommon” isn’t in the vocabulary. A character-level split returns [“u”, “n”, “c”, “o”, “m”, “m”, “o”, “n”], which loses morphological context and creates long sequences.
WordPiece Approach: The algorithm uses a longest-match-first strategy (maximum matching) to split words into subwords. Starting from the beginning of “uncommon,” it looks for the longest subword in the vocabulary. It might first check if “uncommon” exists as a whole word, if not, it continues. It then checks if “un” exists (yes, as a common prefix), marks it as a token, and repeats for the remaining characters “common” (which likely exists as a standalone word). The result is [“un”, “##common”].
Why It’s Effective: Compared to purely word-level tokenization, we avoid <unk> for rare words. Compared to character-level, we keep fewer tokens while retaining morphological insight. These tokens mirror real morphological components: “un” is a common negation prefix, and “common” is a standalone root word. During generation or decoding, the model easily reconstructs “uncommon” by concatenating the tokens. This is central to WordPiece’s strength in large-scale NLP models.
Domain & Multilingual Considerations
Multilingual Scenarios
Models like Multilingual BERT use a combined WordPiece vocabulary for over 100 languages. This can lead to interesting merges (e.g., partially shared subwords across Romance languages) but can also dilute coverage for niche languages or scripts not well represented in the training corpora.
- Shared Vocabulary: Allows for cross-lingual transfer learning and shared representations across languages
- Coverage Trade-offs: Some languages may receive less optimal tokenization due to vocabulary sharing. Low-resource languages or scripts not well-represented may have poor coverage.
Domain Shifts
General-purpose corpora like Wikipedia might not contain domain-specific expressions (medical, legal, scientific jargon). If you plan to parse or generate text in a specialized area, consider:
- Domain Fine-Tuning: Continue training on domain-specific data (e.g., biomedical text for PubMedBERT, legal documents for LegalBERT). Can add domain-specific tokens to the vocabulary.
- Custom Vocabulary: Training WordPiece specifically on domain-specific data from scratch to ensure optimal coverage of domain terminology
WordPiece’s likelihood-based approach can handle new words by sub-splitting them into known subwords, but domain adaptation typically yields better performance and fewer partial subwords. For highly specialized domains, custom tokenization may be necessary.
Strengths, Pitfalls, and Best Practices
Strengths
- Proven Success: Widely used in BERT-based models, demonstrating excellent performance on tasks from question answering to text classification
- Effective Handling of Rare Words: Minimizes <unk> occurrences by splitting words into subword pieces intelligently
- Good Morphological Awareness: The “##” markers and likelihood-based merges often align well with prefixes, suffixes, and linguistic morphemes in many languages
- Robust Ecosystem: Extensive support in popular frameworks and libraries (Hugging Face, TensorFlow)
- Deterministic Tokenization: Longest-match-first strategy produces consistent results
Pitfalls
- Complex Training: The likelihood-based scoring is more computationally intensive than BPE’s frequency counting. No official implementation from Google exists.
- Greedy Tokenization: Longest-match-first doesn’t guarantee globally optimal tokenization for every word
- Fixed Vocabulary: Once trained, the vocabulary can’t adapt to new domains without retraining
- Bias and Coverage: If your training data skews heavily toward certain languages or domains, the resulting merges might not generalize to new text
Best Practices
- Tune Vocab Size: Common values range from ~8k to 64k, depending on data size and domain complexity. BERT-Base uses approximately 30k tokens.
- Evaluate Out-of-Domain Performance: Test your tokenizer on text that might differ from the training domain to gauge subword coverage
- Consider Domain Adaptation: Especially if your data is noisy or morphological variants are common, or for specialized domains
- Leverage Off-the-Shelf Solutions: Frameworks like Hugging Face Transformers offer stable, well-optimized WordPiece tokenizers for immediate use. For most applications, using existing BERT-style tokenizers is the practical choice.
- Monitor Token Length: Track average tokens per word in your domain. If too high, consider domain-specific training.
Conclusion & Future Directions
WordPiece stands as a cornerstone of subword tokenization, offering a likelihood-driven, morphologically-aware, and widely-adopted approach. While BPE and SentencePiece have their own strengths, WordPiece’s scoring mechanism and special markers make it particularly effective for languages with clear word boundaries and Transformer-based architectures. This has made it the de facto standard for models like BERT, DistilBERT, Electra, and their variants.
The key innovation lies in the scoring formula: score(a, b) = freq(ab) / (freq(a) × freq(b)), which prioritizes linguistically meaningful merges over simply the most frequent pairs. This likelihood-based approach, combined with the longest-match-first tokenization strategy, produces vocabularies that balance efficiency with morphological awareness.
Comparisons to BPE and SentencePiece reveal that each method has trade-offs: BPE’s simpler frequency-based merging is faster and easier to implement, SentencePiece excels with multi-script data and languages without clear word boundaries, and WordPiece provides excellent morphological awareness with proven success in Transformer models. In practice, the optimal choice often depends on your language mix, model architecture, and performance requirements.
Looking ahead, some researchers are exploring token-free or continuous representations that bypass subword dictionaries entirely, adaptive tokenization methods that adjust based on context, and improved multilingual vocabulary sharing strategies. Nevertheless, WordPiece remains a practical, proven solution for subword tokenization. Especially if you’re working with Transformer-based models and want morphological awareness with robust ecosystem support. If you’re building NLP models for languages with clear word boundaries or fine-tuning existing Transformer architectures, WordPiece is an excellent choice, it might just optimize both accuracy and efficiency in your language tasks.
Video Explanation
- This video titled, “Wordpiece Tokenization” by Hugging Face, walks through how WordPiece works end-to-end, how the vocabulary is built during training, and how inference does greedy subword matching to tokenize new text (Runtime: 4 mins)
- This video titled, “LLM Tokenizers Explained: BPE Encoding, WordPiece and SentencePiece”, by DataMListic, talks about all the three tokenizers that are commonly used when training large language models: (1) the byte-pair encoding tokenizer, (2) the wordpiece tokenizer and (3) the sentencepiece tokenizer. (Runtime: 5 mins)
