Why your LLM needs an eval set before launch

Almost every LLM feature starts the same way: a few prompts tried by hand, a demo that looks great, and a launch date. Then the first real users arrive and ask things nobody tried. Answers get longer, or vaguer, or confidently wrong — and the team has no way to say whether last week’s prompt change helped or hurt.

An evaluation set is the fix. It is the closest thing an AI product has to a test suite, and it is far cheaper to build before launch than after.

The demo trap

A demo shows that the system can produce a good answer. It says nothing about how often it does, which inputs break it, or what happens when you swap the model. Without a fixed set of cases you are comparing vibes: the last five things you happened to try against the five things you tried yesterday.

What an eval set actually is

At its simplest, an eval set is a versioned list of inputs, plus a way to judge each output. The judging can be an exact check (did it call the right tool?), a metric (is the answer grounded in the retrieved documents?) or a rubric applied by a person or an LLM judge.

{
  "id": "refund-policy-017",
  "input": "Can I return a gift without the receipt?",
  "expected_behaviour": "Cites the gift-return policy; offers store credit; no invented time limits",
  "tags": ["policy", "edge-case"]
}

Building your first one

1. Start from real usage. Pull questions from support tickets, sales calls or beta logs, not from your own imagination.
2. Cover the shape of the problem. Mix common cases, rare-but-important ones and deliberately awkward inputs.
3. Write down what “good” means. Two or three criteria per case are enough to start.
4. Run it on every change. Prompt edits, model swaps and retrieval tweaks all go through the same set.

How big should it be?

Smaller than you think, to start. Fifty well-chosen cases that reflect real traffic will catch more problems than a thousand synthetic ones nobody has read. Grow it every time a bug reaches production: each incident becomes a new case, so the same failure can never ship twice.

The payoff

With an eval set in place, “is this better?” stops being a debate and becomes a number. You can change models with confidence, show stakeholders measurable progress, and catch regressions before your users do.