tokenization-sentencepiece

Topics in the source

• SentencePiece - Language-Independent Tokenization
• When to use SentencePiece
• Quick start
• Installation
• Train model
• Encode and decode
• Language-independent design
• Whitespace as symbol (▁)

Unsupervised tokenizer that works on raw text without language-specific preprocessing. Use SentencePiece when: Building multilingual models (no language-specific rules) Working with CJK languages (Chinese, Japanese, Korean) Need reproducible tokenization (deterministic vocabulary) Want to train on raw text (no pre-tokenization needed) Require lightweight deployment (6MB memory, 50k sentences/sec) Performance: Speed: 50,000 sentences/sec Memory: 6MB for loaded model Languages: All (language-independent) Use alternatives instead: HuggingFace Tokenizers: Faster training, more flexibility tiktoken: OpenAI models (GPT-3.5/4) BERT WordPiece: English-centric tasks Training time: 1-2 minutes for 100MB corpus Key principle: Treat text as raw Unicode, whitespace = ▁ (meta symbol) Used by: mBART Used by: T5, ALBERT, XLNet | Language Type | Coverage | Rationale | |---------------|----------|-----------| | English | 0.9995 | Most common chars | | CJK (Chinese) | 1.0 | All characters needed | | Multilingual | 0.9995 | Balance | Use case: Data augmentation for robustness. | Corpus | BPE (16k) | Unigram (8k) | |--------|-----------|--------------| | 100 MB | 1-2 min | 3-4 min | | 1 GB | 10-15 min

This is an excerpt. Refer to the complete source and its current requirements.