RAG evaluation: the harness I actually run

Every RAG checklist tells you to build a labelled eval set and score answers with an LLM judge. Mine does too. That advice is correct and roughly useless on its own, because the hard parts are all in the words it skips: which queries, labelled how, and why you believe the judge.

This is the workflow rather than the checklist item. It's also the piece that decides whether every other change you make — chunking, swapping the embedding model, reranking — is something you can measure or something you're guessing at.

Where the queries come from

The single decision that determines whether a harness is worth anything: your eval queries have to be real user queries, and they have to be sampled deliberately.

Synthetic queries generated by asking a model to write questions about your documents produce a set that looks fine and tests nothing. The model writes questions your documents obviously answer, phrased the way your documents phrase them. Real users write half-sentences with internal jargon and product names spelled three ways.

So pull from logs. But don't sample randomly — random sampling gives you fifty variations of your most common query and none of the long tail where the failures live. Stratify:

• Head queries — the handful of things most people ask. Regressions here are the ones that generate complaints.
• Long tail — narrow, specific, low-frequency. Where retrieval quality actually varies.
• Known failures — queries you've already seen go wrong. These are the highest-value rows in the set and the ones a random sample will miss entirely.
• Refusals — questions your corpus genuinely cannot answer. If these aren't in the set, nothing measures whether the system says "I don't know" or invents something.

That last category is the one most teams skip, and it's the one that catches the failure mode users find most alarming.

Labelling: two different questions

A single "is this good?" label conflates two systems that fail independently, and averaging them hides which one broke.

For retrieval, the label is a set of chunk or document IDs that should come back. Then recall@k is arithmetic, and it's stable — the label doesn't change when you change your prompt. This is the layer where you'll do most of your iteration, and it's cheap to score exactly.

For generation, the label is a rubric, not a target string. Faithful to the retrieved context, answers the question asked, admits what it doesn't know. Scoring against an exact expected answer breaks the moment the model phrases something differently, and you'll spend your time maintaining the labels rather than the system.

Keeping these separate is what lets you answer "did retrieval get worse, or did the prompt get worse?" — a question that's almost impossible to untangle from a single blended score.

On size, since this is the part that gets deferred indefinitely: fifty queries is the floor, two hundred is comfortable, and beyond that you are mostly buying precision you will never act on. Split them roughly evenly across the four categories rather than in proportion to traffic — your head queries are over-represented in the logs by definition, and they are the least informative slice once they pass.

Budget a day for the first pass, and expect to throw away the first twenty rows. Not because labelling is slow, but because writing labels is how you discover that your rubric is ambiguous, and you will want to redo everything you scored before you understood that. That rewrite is the valuable part — the set you finish with is a better set than the one you set out to build, and the rubric it produces is the thing you will hand to the judge.

Calibrating the judge

Here's the step almost every write-up omits. LLM-as-judge is presented as the solution to scoring generation, and then nobody checks whether the judge is any good.

Measure the judge against yourself before you trust it. Label thirty or so answers by hand, run the judge over the same thirty, and look at the agreement. Not the average score — the per-item agreement, and specifically where it disagrees.

The disagreements are informative in a way the score never is. A judge that's systematically generous on faithfulness will happily pass hallucinations, and its aggregate number will look stable and reassuring while quality drops. A judge that's harsh on brevity will push you toward padded answers users don't want.

Two things that consistently help:

• Make the rubric concrete and few. Three criteria with explicit failure examples beats seven abstract qualities. "Contains a claim not supported by the retrieved chunks" is judgeable; "accuracy" is not.
• Ask for the reason before the score. A judge that explains first and scores second is easier to debug — and when it disagrees with you, the explanation tells you whether the rubric or the system is wrong.

Re-check agreement whenever you change the judge model. A provider upgrading the model underneath you changes your measuring instrument, which is a strange thing to discover by noticing your scores moved.

Running it: a gate and a monitor

The harness does two jobs, and they need different thresholds.

As a gate, it runs on every change to a prompt, a retrieval parameter, or the model — before merge, like a test. The threshold is relative: this change must not reduce recall@5, and must not regress more than N individual queries. Absolute scores are the wrong gate; a change that lifts the mean while breaking six specific queries should fail.

As a monitor, it runs on a schedule against production traffic, because the corpus and the query distribution both drift underneath a static eval set. Documents get added, users start asking about a new feature, and a set that was representative in March is measuring something else by June. Alert on the trend, not on individual scores.

The two feed each other: production failures become labelled rows in the set, which makes the gate stricter over time. That's the flywheel — the harness gets better precisely because the system failed in ways you hadn't imagined. The set itself needs the same discipline applied to it — a flywheel only turns if you notice when the thing it's built on has quietly drifted.

What this costs

It's a real investment. Sampling and labelling a first set is a day or two of unglamorous work, and it will be out of date within months.

The alternative is shipping retrieval changes on vibes, which works until the first change that improves your demo query and quietly degrades the twenty you didn't check. At that point you have no way to tell whether last month's system was better, because you never measured it.

If you're standing up retrieval and this is the part that keeps getting deferred, it's the part I'd do first.