Swap an embedding model without breaking it

A better embedding model comes out, the benchmark numbers are convincingly higher, and swapping looks like a migration script: re-embed the corpus, write the new vectors, point the retriever at the new index. The script is the easy part, and it is not the part that breaks.

What breaks is subtler: aggregate retrieval quality goes up, a specific slice of queries silently gets worse, and nobody notices until a user complains about a question the old system answered fine last month.

The shape of it is consistent enough to predict. A model that wins on public benchmarks lifts your mean recall and buries a regression in the queries carrying your domain's own vocabulary — product names, internal shorthand, the abbreviations your users type and your documentation spells out. Those are precisely the queries a general-purpose benchmark has no opinion about, and precisely the ones your users ask most confidently.

Why the new model breaks queries the old one handled

Two embedding models produce two different vector spaces. That sounds obvious, and its consequences are routinely underestimated.

Similarity scores don't transfer. A cosine similarity of 0.82 in the old model's space and 0.82 in the new one mean different things. Every hard threshold in your code — the score > 0.75 filter that drops weak matches, the cutoff that decides when to say "I don't know" — was tuned against a distribution that no longer exists. The threshold doesn't error. It just starts including or excluding the wrong things.

You cannot mix the two. Nearest-neighbour search across vectors from different models returns noise, not degraded results. That means no incremental migration: the corpus is re-embedded completely, or the index is meaningless. If your ingestion pipeline keeps writing while you backfill, you need to know which model wrote each row.

The failure is uneven. This is the part that catches people. A model that wins on MTEB by three points is not three points better on every query — it's substantially better on some and worse on others. Averages hide that completely. A swap can lift mean recall while regressing the exact narrow, jargon-heavy queries your actual users ask, because public benchmarks aren't built from your corpus.

Build the golden set before you touch the index

You cannot detect an uneven regression without a fixed set of queries you can score both ways. If you already have a labeled eval set, this is what it's for. If you don't, the swap is the wrong time to find out.

What matters more than size is that it's per-query, not aggregate. Score the old model and the new model on the same queries, then diff them one by one. The number you care about isn't mean recall@5 — it's how many queries got worse, and which ones. A swap that improves the mean by four points while regressing 15% of queries is usually not a swap you want, and a summary statistic will never tell you that.

Two things worth capturing while you're there:

• Score retrieval separately from the reranked output. A reranker sitting on top-50 will paper over a retrieval regression right up until the correct document falls out of the top 50 entirely — at which point no amount of reranking recovers it. If you only measure the final answer, you'll miss the retrieval degradation until it's severe.
• Keep the queries that were already failing. They're the ones most likely to flip, in either direction, and they're the cheapest signal you have about whether the new model understands your domain differently.

On size: fifty query/document pairs is the floor I'd argue for anywhere, and it is genuinely enough to catch an uneven regression. You are looking for a directional signal about which queries flipped, not a statistically significant effect — and a set small enough to read end to end is worth more than one large enough to only ever look at in aggregate.

The mix matters far more than the count. If every pair is a clean, well-formed question, the set will report that the swap went fine regardless of what the swap actually did. Weight it toward the queries that already embarrass you: the jargon-heavy ones, the ambiguous ones, the ones where a product name appears spelled three different ways.

And record the regressions individually rather than as a percentage. "Twelve percent got worse" is a number people argue about in a meeting. "These six queries stopped finding the pricing document" is a decision that makes itself.

Shadow the swap against live traffic

A golden set tells you about the queries you thought to write down. Live traffic tells you about the rest.

The pattern worth the extra infrastructure: build the new index alongside the old one, keep serving from the old one, and run every real query through both. Log both result sets. You're not comparing answers yet — you're comparing which documents came back, for queries nobody anticipated.

This costs you a duplicate index and a second embedding call per query for as long as it runs, which is the cheapest insurance in the pipeline. Let it run long enough to cover a realistic slice of your query distribution — including whatever weekly or monthly pattern your traffic has — then compare overlap: for what fraction of live queries did the top-5 change, and when it changed, did it change toward better documents?

Cut over only when the shadow data agrees with the golden set. When they disagree, the golden set is usually the one that's wrong — it's a sample of what you imagined users would ask.

What else moves when the vectors do

The swap touches more than the retriever. The things that quietly come along with it:

• Dimensionality. Going from 1536 to 3072 dimensions doubles storage and slows every query; going down speeds things up and may cost recall. If you're on pgvector, changing dimensions means a new column and a rebuilt index, not an UPDATE — and which store you're on determines how painful that rebuild is.
• Distance metric. Some models are trained for cosine, others for inner product, and some expect normalized vectors. A mismatched metric doesn't error — it just retrieves worse, forever, and it's nearly invisible in code review.
• Asymmetric prefixes. Several strong open models (the E5 and BGE families among them) expect query: and passage: prefixes on the two sides. Omit them, or apply them inconsistently between ingestion and query time, and recall drops sharply for a reason that looks nothing like a prefix bug.
• Chunk size. Models differ in how gracefully they handle long inputs, so a chunking strategy tuned against the old model isn't automatically right for the new one. Worth re-checking on the same golden set rather than assuming it carries over.
• Cost and latency. Both change, in both directions, and the embedding call is now on your ingestion critical path for every backfill.

The rule I follow

Treat an embedding swap as a retrieval-quality change that happens to require a migration, not a migration that happens to touch retrieval. The script is a day. The evaluation is the work.

If the model is genuinely better for your corpus, the golden set and the shadow run will both say so, and you'll cut over knowing which queries you traded away. If you can't tell, you don't yet have the instrumentation to make the call — and that's a more useful thing to learn in a shadow deployment than in a support ticket.

Getting this instrumentation in place before the swap, rather than after the regression, is a large part of what I do.