Embeddings and semantic search

Semantic search finds texts with similar meaning by turning texts into embedding vectors and returning the ones whose vectors are closest to the query's vector.

Why it matters

Keyword search fails when users and documents use different words for the same thing: "can't sign in" versus "forgot my login". Embedding-based search matches meaning, which makes help centres, document search and RAG far better. It is also the retrieval half of almost every LLM application.

The idea

An embedding model turns text into a vector. Texts with similar meaning get similar vectors, even with different words. Search means: embed the question, then find the closest document vectors with cosine similarity.

A library shelved by topic, not by title. 'How do I reset my password' sits next to 'I forgot my login'.

Meaning-based search can miss exact things like product codes or names. That is why many systems combine it with keyword search (hybrid search).

How it works

  1. Choose an embedding model (a text encoder). It maps any text to a fixed-length vector, for example 384 or 1,024 numbers.
  2. Embed every document (or chunk) once, ahead of time, and store the vectors, usually normalised to length 1.
  3. At query time, embed the query with the same model.
  4. Score each document by cosine similarity with the query: with unit vectors, that is just a dot product.
  5. Return the top k documents. A matrix multiply scores thousands of documents in one step.
  6. For exact terms (codes, names), combine with keyword search such as BM25: hybrid search, merging the two rankings.

Key points

Common mistakes

Not true: Semantic search is reliable for exact product codes, IDs and names.
In fact: Embeddings capture meaning, not exact strings, so a code like SKU-4471 can be missed. Combine semantic search with keyword search for those.

Not true: A whole long document is best embedded as one vector.
In fact: One vector for a long document blurs its details into an average. Split the document into chunks and embed each chunk.

Part of Unit 5: AI engineering in the free ML Foundations course.