Token embeddings and positional encoding

Token embeddings turn each token id into a learned vector, and positional encodings add information about where each token sits, because attention on its own ignores order.

Why it matters

Attention compares every token with every other token, but it treats the input as an unordered set: shuffle the words and each output just moves with its word. "dog bites man" and "man bites dog" would look the same. Position information fixes that, and how you encode positions decides how well a model handles long contexts.

The idea

Each token id becomes a vector (its embedding). Attention on its own does not know word order, so we add position information: a learned position vector, sine waves, or rotations (RoPE).

Seat numbers in a theatre. Everyone is a person (the embedding), but the seat number (the position) tells you who sits next to whom.

Seats are fixed. A model trained on 1,000 positions often struggles at 10,000, so long-context models need special tricks.

How it works

  1. Look up each token id in the embedding table: a vector of size d_model (768 in GPT-2 small).
  2. Make a position vector for each position 0, 1, 2, … of the same size.
  3. Learned positions: a second table with one row per position, trained like any weight (GPT-2).
  4. Sinusoidal positions: fixed sine and cosine waves of different frequencies, so every position gets a unique pattern (the original transformer).
  5. Add the two: x = token_embedding + position_embedding. The sum goes into the first transformer block.
  6. RoPE (rotary embeddings, used by Llama and many others) works differently. It rotates each query and key by an angle that depends on its position, so attention scores depend on the relative distance between tokens.

Key points

Common mistakes

Not true: Position information reaches the model as a separate input, kept apart from the token embeddings.
In fact: Learned and sinusoidal positions are simply added to the token embeddings, and rotary positions are applied inside attention. There is no separate position input.

Not true: A model works just as well on texts longer than the context length it was trained on.
In fact: Learned positions do not exist beyond the longest one seen in training, and quality drops quickly past it. Stay within the trained context length.

Part of Unit 4: Transformers and LLMs in the free ML Foundations course.