Re-ranking: the highest-leverage 50 lines

The checklist states this as a bare claim: reranking on top-50 down to top-5 routinely beats raw retrieval by 10+ recall points. This is the post that earns it — what a reranker is actually doing differently from the retrieval step in front of it, and the small amount of code that captures most of the benefit.

Who this is for: an engineer with retrieval already working, ranking by embedding similarity alone, who hasn't added a reranking step because it sounds like a second retrieval system rather than 50 lines on top of the one they have.

Why a reranker beats the retriever at the same task

The embedding model that does retrieval is a bi-encoder — it scores the query and each document independently, encoding both into the same vector space so a similarity comparison is a single dot product. That's what makes it fast enough to run over an entire corpus: precompute every document's embedding once, and a query at runtime only needs its own embedding plus a nearest-neighbor lookup.

A reranker is a cross-encoder — it takes the query and one candidate document together, as a single input, and scores that specific pair directly. It sees interactions between query terms and document terms that a bi-encoder's independent encoding structurally can't capture, because the bi-encoder never lets the two texts influence each other's representation. That's also why a cross-encoder can't run over an entire corpus: scoring is per-pair, so it's only affordable over a small candidate set.

The two are built for different jobs. Retrieval is built to be fast over everything. Reranking is built to be accurate over fifty candidates. Using the retriever's ranking as the final answer is asking the fast, approximate tool to do the accurate tool's job.

The 50 lines

from cohere import Client
 
co = Client(api_key=API_KEY)
 
def rerank(query: str, candidates: list[dict], top_n: int = 5) -> list[dict]:
    docs = [c["text"] for c in candidates]
    response = co.rerank(
        query=query,
        documents=docs,
        top_n=top_n,
        model="rerank-english-v3.0",
    )
    return [candidates[r.index] | {"rerank_score": r.relevance_score} for r in response.results]

Retrieve top-50 by embedding similarity, pass those 50 to the reranker, keep the top 5 it returns for generation. The reranker never sees the corpus — only the candidates retrieval already narrowed it to, which is exactly the division of labor that makes both steps affordable.

Score retrieval and reranking separately

This is the part the checklist calls out explicitly, and it's the part that gets skipped: measure recall@k on the raw retrieval output and again on the reranked output, as two separate numbers. A reranker hides a retrieval regression right up until the correct document falls out of the top 50 entirely — at which point no amount of reranking recovers it, because the document it needed to rerank was never in the candidate set. If you only measure the final answer, retrieval quality can degrade for weeks before anything downstream notices.

Top-50, not top-1000

The candidate count is a real trade, not an arbitrary number. Every candidate the reranker sees is a cross-encoder call, which is why reranking the full result set defeats the point — at 1,000 candidates you've paid for accuracy on a set almost as expensive to score as retrieval was to search. Top-50 is the range where reranking recovers most of the available lift without the per-query cost approaching retrieval's own.

If your eval set shows recall@50 catching the right document reliably but recall@5 missing it, that's the specific signal reranking is built to fix — the answer exists in your candidate set, ranking just isn't surfacing it.

When it doesn't help

If retrieval's top-5 is already close to its top-50 in quality — meaning the bi-encoder is already doing a good job separating relevant from irrelevant on your corpus — reranking buys little because there's little ordering left to fix. That's rarer than teams assume, but it happens on corpora with strong lexical structure where hybrid search already does most of the separating work before reranking would get a turn.

Reranking is close to free to try and cheap to measure: run it against your eval set, diff recall@5 before and after, and the number tells you whether it's worth keeping on for your corpus specifically.

If retrieval is working but the top few results are consistently the wrong ones, reranking is usually the fastest fix available.