Token embeddings and positional encoding
Token embeddings turn each token id into a learned vector, and positional encodings add information about where each token sits, because attention on its own ignores order.
Why it matters
Attention compares every token with every other token, but it treats the input as an unordered set: shuffle the words and each output just moves with its word. "dog bites man" and "man bites dog" would look the same. Position information fixes that, and how you encode positions decides how well a model handles long contexts.
The idea
Each token id becomes a vector (its embedding). Attention on its own does not know word order, so we add position information: a learned position vector, sine waves, or rotations (RoPE).
Seat numbers in a theatre. Everyone is a person (the embedding), but the seat number (the position) tells you who sits next to whom.
Seats are fixed. A model trained on 1,000 positions often struggles at 10,000, so long-context models need special tricks.
How it works
- Look up each token id in the embedding table: a vector of size d_model (768 in GPT-2 small).
- Make a position vector for each position 0, 1, 2, … of the same size.
- Learned positions: a second table with one row per position, trained like any weight (GPT-2).
- Sinusoidal positions: fixed sine and cosine waves of different frequencies, so every position gets a unique pattern (the original transformer).
- Add the two: x = token_embedding + position_embedding. The sum goes into the first transformer block.
- RoPE (rotary embeddings, used by Llama and many others) works differently. It rotates each query and key by an angle that depends on its position, so attention scores depend on the relative distance between tokens.
Key points
- Without positions, attention is blind to order.
- Learned positions can't go beyond the longest position seen in training.
- Sinusoidal and rotary schemes generalise further, and can be stretched for longer contexts.
- RoPE encodes relative position directly in the query·key dot product.
- Embeddings and positions are both d_model wide, so they can simply be added.
- Many models tie the input embedding table to the output layer (weight tying) to save parameters.
Common mistakes
- Forgetting positions entirely in a hand-built transformer.
- Running a model past its trained context length and expecting good results.
- Adding positions with the wrong shape, so they broadcast across the batch instead of the sequence.
Not true: Position information reaches the model as a separate input, kept apart from the token embeddings.
In fact: Learned and sinusoidal positions are simply added to the token embeddings, and rotary positions are applied inside attention. There is no separate position input.
Not true: A model works just as well on texts longer than the context length it was trained on.
In fact: Learned positions do not exist beyond the longest one seen in training, and quality drops quickly past it. Stay within the trained context length.
Part of Unit 4: Transformers and LLMs in the free ML Foundations course.