Evaluation: test sets, LLM-as-judge, error analysis
Evaluation measures whether an LLM system does its job. It uses a fixed test set of real inputs, automatic checks or judges for scoring, and careful reading of the failures to find what to fix.
Why it matters
LLM apps change constantly — prompts, models, retrieval settings — and every change can quietly break something else. Without evals you are guessing from a few hand-picked examples. A good eval set turns "it feels better" into a number you can compare, and error analysis tells you which fix will help most.
The idea
Evaluation tells you whether a change made your system better or worse. Build a test set of real inputs with expected results. Score with code checks where you can, and with an LLM judge where you must. Most important: read the failures and group them into error types.
Unit tests for software, but for behaviour that is fuzzy.
Unit tests pass or fail. LLM outputs need judgement, and the judge (human or model) can be wrong, so check the judge against human labels.
How it works
- Collect 50–200 real inputs that cover the cases you care about, including hard and edge cases, each with an expected answer or a grading rule.
- Score with code wherever possible: does the JSON parse, is the number right, was the right document cited?
- Where judgement is needed, use an LLM judge with a precise rubric, and check it against your own labels on a sample.
- Run the whole system on the set and record the score and every output.
- Read the failures one by one and label why each failed; group the labels into error types.
- Fix the biggest error type first, rerun the same set, and compare.
- Keep adding new failures from real use to the set, so the same bug can't return unnoticed.
Key points
- Evals for LLM apps are like unit tests, but graded on fuzzy behaviour.
- Code checks first; LLM judges where you must; humans to check the judges.
- Error analysis (reading failures) is the most valuable step.
- Compare versions on the same test set.
- Measure the parts (retrieval, tool choice) as well as the end result.
- Watch cost and latency alongside quality.
Common mistakes
- Testing on a handful of examples you already know work.
- Trusting an LLM judge you have never checked against people.
- Changing the test set and the system at the same time.
- Tracking one overall score without reading any failures.
Not true: One overall score shows what to fix in an LLM app.
In fact: A score says how often the app fails, not why. Reading the failures and grouping them into types shows what to fix first.
Not true: An LLM judge can be trusted without ever comparing it with human judgement.
In fact: A judge model has its own blind spots and biases. Check a sample of its verdicts against people before relying on it.
Part of Unit 5: AI engineering in the free ML Foundations course.