sdlc.dev

AI engineering path · Shipping with confidence · Topic 9 of 12

Evaluating LLM features

Not started

An evaluation (eval) is a test suite for behaviour that has no single correct string: a fixed set of inputs, a way to grade each output, and a score you can compare between versions. It is what lets you change a prompt, a retrieval setting or a model and say with evidence whether things improved. Teams that ship LLM features reliably usually have evals before they have clever prompts.

Why it matters: Without evaluations every prompt or model change is a guess checked by eye on three examples. With them, you can change things quickly and know what got better and what broke.

  1. 1A golden set of real cases

    Collect realistic inputs, including edge cases and past failures, each with the expected answer or the criteria a good answer meets.

  2. 2Cheapest reliable grader first

    Exact match, schema checks and code-based rules where possible; a rubric-guided model grader where judgement is needed; humans to calibrate both.

  3. 3Compare, don't just score

    Run old and new versions on the same set and look at which cases flipped, not only the average.

  4. 4LLM judges have biases

    Model graders can favour longer answers, the first option shown, or their own style. Check them against human labels.

1

How it works

The explanation, with examples you can run.

Build the golden set

Start small and real: 20 to 50 cases drawn from actual or realistic usage, then grow it. For each case store the input, and either the expected output or the criteria a good output must meet. Deliberately include:

  • typical requests, in proportion to real traffic,
  • edge cases: empty input, very long input, other languages, ambiguous questions, questions your system should decline,
  • every bug you fix: a production failure becomes a permanent test case.

Keep the set in version control. When you change the expected answers, review the change like code.

Choose graders

Use the simplest grader that is reliable for each criterion:

  • Code-based checks: exact match for labels, schema validation, "contains the order id", "under 120 words", "every cited passage id was supplied", "no forbidden phrases". Fast, cheap, deterministic.
  • Model-based grading (LLM-as-judge): a model scores the output against a written rubric ("Does the reply answer the customer's question using only the policy? Score 1 to 5 and give a one-line reason"). Useful for tone, faithfulness and completeness, where code cannot judge.
  • Human review: slow and expensive, so use it to build the golden answers, to calibrate model graders, and to spot-check.

Model graders need care. Studies of LLM judges report position bias, verbosity bias and a tendency to prefer answers resembling their own. Ask for one criterion at a time, give an explicit rubric with examples of each score, swap the order in pairwise comparisons, and measure how often the grader agrees with human labels before trusting it.

Compare versions case by case

The run below grades two versions of a ticket classifier (fake functions standing in for prompts) on the same golden set, then lists the cases that flipped. The average alone would hide the regression.

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

Make it part of the workflow

  • Run evals on every change to prompts, retrieval settings, tools or model version, in CI if the set is small enough, nightly if not. Treat a drop below a threshold, or a regression on a case marked critical, as a failing build.
  • Run each case more than once when outputs vary, and report pass rates; with a small set, a one-case difference may be noise.
  • Evaluate stages separately in pipelines: retrieval recall, then answer quality given correct context, then the end-to-end result. It tells you where to fix.
  • Watch production too. Offline sets drift away from real usage. Sample live traffic (with appropriate consent and privacy controls), collect user feedback such as thumbs down and corrections, and feed representative cases back into the golden set.
2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Your new prompt raises the average score from 82% to 85%, but four cases that used to pass now fail. What should you do?
  2. 2.You use a model to compare two answers and pick the better one. Which precaution addresses a known bias of LLM judges?
  3. 3.Why run each eval case more than once?

0 of 3 right so far. Get all three to finish this topic.

Go deeper