Tokenization
How text becomes the integer ids a language model reads, why subword pieces won, and the odd behaviour that comes from it.
Why tokens exist
A neural network reads numbers, not letters. A tokenizer converts text into a list of integer token ids and back. A token is a piece of text: sometimes a whole word, sometimes part of a word, sometimes one character or byte.
Three ways to cut text:
| Unit | Vocabulary size | Problem |
|---|---|---|
| Whole words | Hundreds of thousands | Unseen words ("untokenizable") have no entry; huge embedding table |
| Single characters | Tens to thousands | Sequences become very long, so attention costs more and meaning is spread thin |
| Subwords | Typically 30,000 to 250,000 | A balance: common words stay whole, rare words split into known pieces |
1 of 7