Tokenisation and byte-pair encoding (BPE)
Tokenisation splits text into the units a language model reads, called tokens. Byte-pair encoding (BPE) builds those units by starting from single characters (or bytes) and repeatedly merging the most frequent neighbouring pair.
Why it matters
A model needs a fixed vocabulary of ids. Whole words give a huge vocabulary and no way to handle new or misspelled words. Single characters make sequences very long and slow. BPE sits in between: common words become one token, rare words break into familiar pieces, and nothing is ever "unknown". Tokens are also how LLM context and cost are measured.
The idea
A model reads tokens, not letters or whole words. Byte-pair encoding (BPE) starts from single characters (or bytes) and repeatedly merges the most frequent neighbouring pair into a new token.
Shorthand notes. You notice you write 'the' all the time, so you invent one symbol for it.
You choose shorthand by meaning. BPE merges only by frequency, so tokens can be odd pieces like 'ing' or ' the' with a space.
How it works
- Start with every character (or every byte, in GPT-style tokenizers) as its own token.
- Count every pair of neighbouring tokens across the training text.
- Merge the most frequent pair into one new token and add it to the vocabulary. In "low lower lowest", "l"+"o" is merged first, then "lo"+"w" makes "low".
- Repeat, tens of thousands of times, until the vocabulary reaches its target size (GPT-2 has about 50,000 tokens).
- To encode new text, apply the learned merges in the same order. To decode, join the tokens' bytes back together.
- Spaces usually travel with the next word, so " the" (with a space) and "the" are different tokens.
Key points
- Token ≠ word. In English, 1 token is roughly 4 characters, or about three-quarters of a word.
- Byte-level BPE can encode any text, emoji included, with no unknown tokens.
- Context length, prices and speed are all counted in tokens.
- Numbers and non-English scripts often split into many tokens, which costs more and makes arithmetic harder.
- The tokenizer is trained once and fixed; the model must always use the same one.
- Vocabulary size trades sequence length against the size of the embedding table.
Common mistakes
- Assuming one token = one word when estimating cost or context.
- Using a different tokenizer from the one the model was trained with.
- Forgetting that leading spaces and capital letters change tokens.
- Blaming the model for odd behaviour on spelling or digit-counting that really comes from tokenisation.
Not true: A word always becomes the same tokens, with or without a space or a capital letter in front of it.
In fact: A leading space or a capital letter changes the tokens: "apple", " apple" and "Apple" are all different. That is why exact spacing matters in a prompt.
Not true: One token is always one word.
In fact: A token is often a piece of a word. In English, one token is about 4 characters, or three-quarters of a word, so a text has more tokens than words.
Part of Unit 4: Transformers and LLMs in the free ML Foundations course.