Why eval sets go stale

The flywheel is the easy half to state and the hard half to run: production failures become new labelled rows, which makes the gate stricter over time. What that sentence skips is how you notice a failure is a failure, once the thing you're comparing it against has quietly stopped matching the world it was built to represent.

Who this is for: an engineer with a working eval harness who hasn't touched the eval set itself in a couple of months, and isn't sure whether that's fine or a problem.

Three things drift, independently

The corpus drifts. Documents get added and removed continuously, and an eval set's queries were written against a specific snapshot of that corpus. A query whose correct answer lived in a document that's since been deleted or superseded doesn't fail loudly — it just quietly stops meaning what it used to mean, and your recall number keeps reporting against a target that no longer exists.

The query distribution drifts. Users start asking about a feature that shipped last month, or stop asking about one that got deprecated. Retrieval recall drops on the new topic and nobody notices for two weeks, because the eval set — built before that topic existed — has no queries that would catch it. A set that was representative in March is measuring a distribution that stopped being real in April.

The judge drifts. If you're scoring generation with an LLM judge, the model underneath it can change without you choosing to change it — a provider upgrades the model behind an API endpoint, and your measuring instrument just moved. Scores shift, and the shift looks like a system regression when it's actually a judge recalibration nobody asked for.

None of these three show up in a single "eval score" number. They show up as the number quietly meaning something different than it did last month, which is a much harder thing to notice than a number going down.

The detection signal for each

• Corpus drift: track the fraction of eval-set source documents that have been modified or removed since the query was written. Above a threshold — even a rough one, 10% is a reasonable start — that query is a candidate for re-verification, not automatic removal, because the document changing doesn't always mean the answer changed.
• Query distribution drift: sample a rolling window of real production queries monthly and diff the topic distribution against the eval set's own. A growing gap between what users ask and what the eval set tests is the signal, not any single query.
• Judge drift: re-run the calibration check — the same 30-answer hand-labelled comparison — whenever a provider announces a model update behind an API you call, not on a fixed schedule. The trigger is the provider's changelog, not your calendar.

The refresh workflow

Weekly re-sampling, not a full rebuild: pull a slice of the last week's production queries, run them through the harness, and hand-label the ones that fail or that the judge scores with low confidence. This is the mechanism that makes the flywheel actually turn — every real failure becomes a labelled row, and the set grows in the direction your actual traffic is moving rather than the direction you guessed it would move in six months ago.

Retire old rows on the same cadence you add new ones. An eval set that only grows becomes slow to run and, worse, lets long-dead queries keep influencing an aggregate score that should reflect your current corpus. A query whose source document has been gone for three months and was never re-verified is weight your gate is carrying for no reason.

What this doesn't need to be

Not a full rebuild on a fixed schedule — quarterly eval-set overhauls are exactly the wrong cadence, because by the time one arrives you've been measuring against a stale target for most of the quarter. Weekly, small, and triggered by real signal beats large and scheduled every time.

If your eval set hasn't changed since you built it and your system has shipped anything since then, it's worth an afternoon to check which of the three drifts already happened.