Embeddings: representing things as vectors
An embedding gives each item, such as a word, a token, a user or a product, a learned vector of numbers, stored as one row of a table. Similar items end up with vectors that are close together.
Why it matters
Networks need numbers, but words and ids are just labels. One-hot vectors are huge, sparse and treat every pair of items as equally different. Embeddings give each item a small dense vector that training shapes to be useful, so "cat" and "dog" end up close. They are the input layer of every language model and the basis of semantic search and recommendations.
The idea
An embedding is a learned vector for each item: a word, a user, a product. Similar items end up with similar vectors. Technically, it is a lookup table with one row per item.
A city map where similar shops end up in the same neighbourhood, so distance means similarity.
A map has 2 dimensions with names like 'north'. Embeddings have hundreds of dimensions, and they rarely have simple names.
How it works
- Create a table E with one row per item: vocabulary size × embedding size, for example 50,000 × 768. Start it random.
- To embed item id i, take row i: E[i]. It is a lookup, the same as multiplying a one-hot vector by E, but much faster.
- Feed the vectors into the rest of the network.
- During training, the gradient flows back into only the rows that were used, and nudges them to be more useful for the task.
- Items used in similar contexts get similar gradients, so their vectors drift together.
- Measure similarity between items with cosine similarity of their vectors.
- Directions can carry meaning: in word2vec, king − man + woman lands near queen.
Key points
- An embedding layer is a learnable lookup table: nn.Embedding(num_items, dim).
- Lookup = one-hot × E, without building the one-hot.
- Only the rows used in a batch get gradients.
- Similar items → nearby vectors; compare them with cosine similarity.
- A single dimension of an embedding rarely has a human-readable meaning: the meaning lies in directions and distances.
- Pre-trained embeddings (from language models) power semantic search and RAG (Unit 5).
Common mistakes
- An id bigger than the table size: an index error.
- An embedding size so small it can't separate items, or so large it overfits small data.
- Comparing embeddings from two different models as if they shared a space.
Not true: Embeddings from two different models can be compared directly, vector against vector.
In fact: Each model learns its own space, with its own directions. Vectors can only be compared inside the model that produced them.
Not true: A bigger embedding size is always better.
In fact: Too small and the vectors can't tell items apart; too large and they overfit small data and waste memory. Choose the size to fit the data.
Part of Unit 3: Deep learning foundations in the free ML Foundations course.