Tokenisation and byte-pair encoding (BPE)

Tokenisation splits text into the units a language model reads, called tokens. Byte-pair encoding (BPE) builds those units by starting from single characters (or bytes) and repeatedly merging the most frequent neighbouring pair.

Why it matters

A model needs a fixed vocabulary of ids. Whole words give a huge vocabulary and no way to handle new or misspelled words. Single characters make sequences very long and slow. BPE sits in between: common words become one token, rare words break into familiar pieces, and nothing is ever "unknown". Tokens are also how LLM context and cost are measured.

The idea

A model reads tokens, not letters or whole words. Byte-pair encoding (BPE) starts from single characters (or bytes) and repeatedly merges the most frequent neighbouring pair into a new token.

Shorthand notes. You notice you write 'the' all the time, so you invent one symbol for it.

You choose shorthand by meaning. BPE merges only by frequency, so tokens can be odd pieces like 'ing' or ' the' with a space.

How it works

  1. Start with every character (or every byte, in GPT-style tokenizers) as its own token.
  2. Count every pair of neighbouring tokens across the training text.
  3. Merge the most frequent pair into one new token and add it to the vocabulary. In "low lower lowest", "l"+"o" is merged first, then "lo"+"w" makes "low".
  4. Repeat, tens of thousands of times, until the vocabulary reaches its target size (GPT-2 has about 50,000 tokens).
  5. To encode new text, apply the learned merges in the same order. To decode, join the tokens' bytes back together.
  6. Spaces usually travel with the next word, so " the" (with a space) and "the" are different tokens.

Key points

Common mistakes

Not true: A word always becomes the same tokens, with or without a space or a capital letter in front of it.
In fact: A leading space or a capital letter changes the tokens: "apple", " apple" and "Apple" are all different. That is why exact spacing matters in a prompt.

Not true: One token is always one word.
In fact: A token is often a piece of a word. In English, one token is about 4 characters, or three-quarters of a word, so a text has more tokens than words.

Part of Unit 4: Transformers and LLMs in the free ML Foundations course.