Evaluation: test sets, LLM-as-judge, error analysis

Evaluation measures whether an LLM system does its job. It uses a fixed test set of real inputs, automatic checks or judges for scoring, and careful reading of the failures to find what to fix.

Why it matters

LLM apps change constantly — prompts, models, retrieval settings — and every change can quietly break something else. Without evals you are guessing from a few hand-picked examples. A good eval set turns "it feels better" into a number you can compare, and error analysis tells you which fix will help most.

The idea

Evaluation tells you whether a change made your system better or worse. Build a test set of real inputs with expected results. Score with code checks where you can, and with an LLM judge where you must. Most important: read the failures and group them into error types.

Unit tests for software, but for behaviour that is fuzzy.

Unit tests pass or fail. LLM outputs need judgement, and the judge (human or model) can be wrong, so check the judge against human labels.

How it works

  1. Collect 50–200 real inputs that cover the cases you care about, including hard and edge cases, each with an expected answer or a grading rule.
  2. Score with code wherever possible: does the JSON parse, is the number right, was the right document cited?
  3. Where judgement is needed, use an LLM judge with a precise rubric, and check it against your own labels on a sample.
  4. Run the whole system on the set and record the score and every output.
  5. Read the failures one by one and label why each failed; group the labels into error types.
  6. Fix the biggest error type first, rerun the same set, and compare.
  7. Keep adding new failures from real use to the set, so the same bug can't return unnoticed.

Key points

Common mistakes

Not true: One overall score shows what to fix in an LLM app.
In fact: A score says how often the app fails, not why. Reading the failures and grouping them into types shows what to fix first.

Not true: An LLM judge can be trusted without ever comparing it with human judgement.
In fact: A judge model has its own blind spots and biases. Check a sample of its verdicts against people before relying on it.

Part of Unit 5: AI engineering in the free ML Foundations course.