Source: Wang et al.
Introduction
Tokenization is a foundational step in natural language processing (NLP), converting raw text into smaller units that machine learning models can process. In languages such as English, tokenization is relatively straightforward because spaces naturally separate words. However, many widely spoken languages, such Chinese, Japanese, Thai, and Khmer, do not use spaces to mark word boundaries. This characteristic introduces ambiguity and complexity, making standard whitespace-based tokenization ineffective. Choosing the right tokenizer is therefore critical for building accurate and efficient NLP systems for these languages.
Subword Tokenization
One of the most effective solutions for languages without spaces is subword tokenization, which is now the dominant approach in modern NLP. Subword tokenizers divide text into smaller units based on statistical patterns rather than predefined word boundaries. Common methods include Byte Pair Encoding (BPE), WordPiece, and Unigram Language Models, often implemented through tools like SentencePiece. These tokenizers are language-agnostic and can operate directly on raw text, making them particularly well-suited for multilingual and transformer-based models. They also handle rare words, compound expressions, and evolving vocabularies more robustly than word-based methods.
Character-Based Tokenization
Another approach is character-based tokenization, where each character is treated as a token. This method is sometimes used for logographic languages like Chinese and Japanese, where characters often convey semantic meaning. While character tokenization avoids the need for word segmentation, it produces much longer sequences and can lose higher-level semantic structure, making it less efficient for large-scale models.
Dictionary-Based or Morphological Tokenizers
A third category includes dictionary-based or morphological tokenizers, which rely on linguistic rules and curated lexicons. Examples include Jieba for Chinese and MeCab for Japanese. These tools aim to produce linguistically meaningful word segments and can be effective in traditional NLP pipelines. However, they are language-specific, require maintenance, and are often incompatible with pretrained transformer models for several reasons. One incompatibility is that transformer models are trained with a specific subword vocabulary, so using a different tokenizer changes the input representation. Additionally, dictionary-based tokenizers produce word-level tokens that do not match the model’s learned embeddings. And finally, segmentation errors or dictionary updates can introduce token mismatches that the model was never trained to handle. As a result, performance can degrade significantly unless the model is retrained from scratch.
Overview
Each tokenization approach comes with its own set of pros and cons:
| Tokenization Approach | Pros | Cons |
|---|---|---|
| Subword Tokenization (BPE, WordPiece, Unigram / SentencePiece) | • Language-agnostic and works without spaces • Handles rare words and new terms well • Efficient vocabulary size • Standard for modern transformer models (BERT, GPT, T5, XLM-R) • Compatible with multilingual training | • Tokens may not align with linguistic words • Less interpretable than word-level tokenization • Requires training a tokenizer on large corpora |
| Character-Based Tokenization | • No need for word segmentation • Simple and robust • Works naturally for logographic scripts • Avoids out-of-vocabulary issues entirely | • Produces very long sequences • Higher computational cost • Loses word-level and subword semantics • Generally lower performance on complex tasks |
| Dictionary-Based / Morphological Tokenization | • Produces linguistically meaningful word units • Good for traditional NLP tasks (POS tagging, parsing) • Shorter sequences than character-based methods | • Language-specific and not portable • Requires curated dictionaries and maintenance • Struggles with new or informal words • Often incompatible with pretrained transformer models |
Source: AIML.com Research
Bottom Line: For most contemporary NLP applications, especially those involving transformer architectures, the best choice for languages without spaces is subword tokenization, particularly using tools like SentencePiece with BPE or Unigram models. This approach balances efficiency, flexibility, and cross-lingual applicability while avoiding the complexities of explicit word segmentation. Ultimately, the most important rule is consistency: always use the same tokenizer that a pretrained model was trained with to ensure optimal performance.
Video Explanations
- This video “Byte Pair Encoding Tokenization” by HuggingFace provides explanation of BPE, one of the tokenization techniques used for languages that are not separated via spaces (Runtime: 6 mins)
- This video “Building Large Language Models (LLMs)” by Stanford Online presents a lecture on LLMs. Starting 8:48, the video explains the importance of tokenization and gives an overview of the tokenization process with an example (Runtime: 7 mins)
Building Large Language Models (LLMs) by Stanford Online