RAG: chunking, retrieval and re-ranking

Retrieval-augmented generation (RAG) answers questions by retrieving relevant pieces of your documents and giving them to the model in the prompt, so it answers from that evidence.

Why it matters

Models don't know your private documents or anything after their training cut-off, and they make things up when they don't know. Fine-tuning facts in is slow, costly and unreliable. RAG keeps knowledge in documents you can update at any time, lets the model cite its sources, and is the most common pattern in real LLM products.

The idea

Retrieval-augmented generation (RAG): find relevant pieces of your documents, put them in the prompt, and ask the model to answer from them. Chunking splits documents into pieces. Re-ranking uses a second, more careful model to re-order the top results.

An open-book exam. The model is the student. Retrieval is finding the right pages before answering.

A student can flip through the whole book. The model only sees the pages you hand it. If retrieval picks the wrong pages, the answer is wrong but still sounds confident.

How it works

  1. Indexing (offline): split documents into chunks of a few hundred words, with a little overlap, following headings where possible.
  2. Embed each chunk and store it in an index with metadata (source, date, section).
  3. At question time, embed the question and retrieve the most similar chunks, often with hybrid keyword + semantic search.
  4. Optionally re-rank: a slower cross-encoder model scores each (question, chunk) pair and re-orders the top 20–50 candidates.
  5. Build the prompt: instructions, the best chunks clearly marked as sources, then the question. Tell the model to answer only from the sources, cite them, and say when the answer isn't there.
  6. The model generates a grounded answer with citations.
  7. Evaluate retrieval (did the right chunk come back?) and generation (is the answer correct and faithful?) separately.

Key points

Common mistakes

Not true: When the sources don't contain the answer, a RAG system says so without being told to.
In fact: Unless the prompt covers that case, the model usually guesses. Tell it plainly to say when the answer is not in the sources.

Not true: Evaluating a RAG system: check the final answers; retrieval needs no separate check.
In fact: A wrong answer can come from retrieving the wrong chunks, or from misusing the right ones. Measure retrieval on its own, so you know which part failed.

Part of Unit 5: AI engineering in the free ML Foundations course.