Choosing between prompting, RAG and fine-tuning
Prompting, RAG and fine-tuning are three ways to adapt an LLM: prompting changes the instructions, RAG supplies knowledge at question time, and fine-tuning changes the model's weights to change its behaviour.
Why it matters
Teams often jump to fine-tuning when a better prompt or retrieval would do, wasting weeks, or try to prompt their way around missing knowledge. Choosing the right tool — or the right combination — saves cost and time and gives better results.
The idea
Try in this order. Prompting first, because it is cheapest. RAG when the model needs knowledge it does not have. Fine-tuning when it needs a new behaviour, style or format that prompts cannot give reliably.
A new employee. First give instructions (the prompt). Give them the company handbook for facts (RAG). Send them on a training course to change how they work (fine-tuning).
Training courses are a poor way to learn facts that change often, and fine-tuning is too. Use RAG for changing knowledge.
How it works
- Start with prompting: clear instructions and examples. Measure on your eval set.
- If the model lacks knowledge (your documents, recent or per-user facts), add RAG or a lookup tool.
- If it has the knowledge but won't reliably follow a behaviour, style or format despite good prompts and examples, consider fine-tuning.
- Fine-tuning can also make a small model do one narrow task as well as a large one, cutting cost and latency.
- Combine them when needed: a fine-tuned model that follows your format can still use RAG for facts.
- Re-measure after each step, and stop as soon as quality is good enough.
Key points
- Order: prompt → RAG (knowledge) → fine-tune (behaviour).
- RAG for facts that change or are private; fine-tuning is poor at storing facts.
- Fine-tuning for format, style, tone and narrow skills.
- Fine-tuning needs data, evals and maintenance; prompts are instant to change.
- Prompting, RAG and fine-tuning are not exclusive: many systems use all three together.
Common mistakes
- Fine-tuning to add knowledge that changes weekly.
- Fine-tuning before trying a good prompt with examples.
- Using RAG to fix a formatting problem.
- Not measuring, so the "improvement" is a guess.
Not true: RAG: the tool for fixing the format of a model's answers.
In fact: RAG supplies knowledge. Format, style and tone are behaviour: fix them with a better prompt and examples first, and with fine-tuning if that is not enough.
Not true: First thing to try when answers fall short: fine-tuning.
In fact: Start with a good prompt and examples: a prompt is instant to change. Add RAG for knowledge, and fine-tune only if the behaviour is still wrong.
Part of Unit 5: AI engineering in the free ML Foundations course.