RAG: chunking, retrieval and re-ranking
Retrieval-augmented generation (RAG) answers questions by retrieving relevant pieces of your documents and giving them to the model in the prompt, so it answers from that evidence.
Why it matters
Models don't know your private documents or anything after their training cut-off, and they make things up when they don't know. Fine-tuning facts in is slow, costly and unreliable. RAG keeps knowledge in documents you can update at any time, lets the model cite its sources, and is the most common pattern in real LLM products.
The idea
Retrieval-augmented generation (RAG): find relevant pieces of your documents, put them in the prompt, and ask the model to answer from them. Chunking splits documents into pieces. Re-ranking uses a second, more careful model to re-order the top results.
An open-book exam. The model is the student. Retrieval is finding the right pages before answering.
A student can flip through the whole book. The model only sees the pages you hand it. If retrieval picks the wrong pages, the answer is wrong but still sounds confident.
How it works
- Indexing (offline): split documents into chunks of a few hundred words, with a little overlap, following headings where possible.
- Embed each chunk and store it in an index with metadata (source, date, section).
- At question time, embed the question and retrieve the most similar chunks, often with hybrid keyword + semantic search.
- Optionally re-rank: a slower cross-encoder model scores each (question, chunk) pair and re-orders the top 20–50 candidates.
- Build the prompt: instructions, the best chunks clearly marked as sources, then the question. Tell the model to answer only from the sources, cite them, and say when the answer isn't there.
- The model generates a grounded answer with citations.
- Evaluate retrieval (did the right chunk come back?) and generation (is the answer correct and faithful?) separately.
Key points
- Retrieval quality caps answer quality: wrong chunks give confident, wrong answers.
- Chunk size is a trade-off: too small loses context, too large blurs relevance and wastes tokens.
- Hybrid search and re-ranking usually give the biggest retrieval gains.
- Add context to chunks (document title, section) so they make sense on their own.
- Citations let users check answers and make failures visible.
- Update knowledge by re-indexing documents, not retraining the model.
Common mistakes
- Only evaluating final answers, so retrieval failures stay hidden.
- Chunking that cuts tables, lists or sentences in half.
- No instruction for "the answer isn't in the sources", so the model guesses.
- Ignoring dates and permissions in metadata.
Not true: When the sources don't contain the answer, a RAG system says so without being told to.
In fact: Unless the prompt covers that case, the model usually guesses. Tell it plainly to say when the answer is not in the sources.
Not true: Evaluating a RAG system: check the final answers; retrieval needs no separate check.
In fact: A wrong answer can come from retrieving the wrong chunks, or from misusing the right ones. Measure retrieval on its own, so you know which part failed.
Part of Unit 5: AI engineering in the free ML Foundations course.