Embeddings and semantic search
Semantic search finds texts with similar meaning by turning texts into embedding vectors and returning the ones whose vectors are closest to the query's vector.
Why it matters
Keyword search fails when users and documents use different words for the same thing: "can't sign in" versus "forgot my login". Embedding-based search matches meaning, which makes help centres, document search and RAG far better. It is also the retrieval half of almost every LLM application.
The idea
An embedding model turns text into a vector. Texts with similar meaning get similar vectors, even with different words. Search means: embed the question, then find the closest document vectors with cosine similarity.
A library shelved by topic, not by title. 'How do I reset my password' sits next to 'I forgot my login'.
Meaning-based search can miss exact things like product codes or names. That is why many systems combine it with keyword search (hybrid search).
How it works
- Choose an embedding model (a text encoder). It maps any text to a fixed-length vector, for example 384 or 1,024 numbers.
- Embed every document (or chunk) once, ahead of time, and store the vectors, usually normalised to length 1.
- At query time, embed the query with the same model.
- Score each document by cosine similarity with the query: with unit vectors, that is just a dot product.
- Return the top k documents. A matrix multiply scores thousands of documents in one step.
- For exact terms (codes, names), combine with keyword search such as BM25: hybrid search, merging the two rankings.
Key points
- Same model for documents and queries, always.
- Cosine similarity: 1 = same direction, 0 = unrelated.
- Normalise vectors once, then dot products are cosine similarities.
- Semantic search handles synonyms and paraphrases; keyword search handles exact strings. Hybrid gets both.
- Quality depends heavily on the embedding model and on how documents are chunked.
Common mistakes
- Mixing embeddings from different models. Each model has its own space, so their vectors can't be compared.
- Relying on semantic search alone for product codes, IDs and names.
- Embedding whole long documents as one vector, so specific details are blurred.
- Not evaluating retrieval separately from the final answer.
Not true: Semantic search is reliable for exact product codes, IDs and names.
In fact: Embeddings capture meaning, not exact strings, so a code like SKU-4471 can be missed. Combine semantic search with keyword search for those.
Not true: A whole long document is best embedded as one vector.
In fact: One vector for a long document blurs its details into an average. Split the document into chunks and embed each chunk.
Part of Unit 5: AI engineering in the free ML Foundations course.