A production RAG checklist (the boring half)

A RAG pipeline is easy to demo and hard to ship. The flashy part — embedding documents, vector search, generation — is roughly 30% of the work. The other 70% is what determines whether you wake up at 3am.

Here's the checklist I run through before calling any RAG system "production." It's the same pass I run when reviewing someone else's — the list doesn't change depending on who built the thing.

Who this is for: an engineer who has a RAG pipeline returning plausible answers and has to decide whether it's safe to put in front of users. If you're still choosing an embedding model, you're earlier than this list.

Every item is phrased so you can answer yes or no. "Sort of" is a no — most of these fail quietly, and a maybe means you haven't looked.

Retrieval quality

• You have a labeled eval set of at least 50 query/answer pairs from real user queries (not synthetic). Questions generated from your own documents are phrased the way your documents are phrased, so they pass easily and prove nothing. Pull from logs, and include queries your corpus genuinely can't answer.
• Retrieval recall@k is measured and tracked over time. If you don't know your recall@5, you don't have a system, you have hope. The tracking matters more than the number — a single measurement tells you where you are, not which direction you're moving.
• You've tested chunking strategies with the actual eval set (paragraph, semantic, sliding-window). Pick the one that wins on recall, not the one that's prettiest.
• Hybrid search (dense + BM25) is in place. Pure vector search loses to hybrid on most real corpora, and it loses hardest on the queries that matter most: product codes, error strings, version numbers, surnames. Embeddings are built to blur near-synonyms, which is exactly wrong when the user typed an identifier.
• Reranking step is configured (Cohere Rerank, BGE reranker, or similar). Reranking on top-50 → top-5 routinely beats raw retrieval by 10+ recall points. Measure retrieval and reranked output separately, though — a reranker hides a retrieval regression right up until the correct document falls out of the top 50 entirely, and then no amount of reranking gets it back.

Generation quality

• You have a rubric-based eval for generated answers (faithfulness, relevance, completeness). Use LLM-as-judge with calibrated prompts, not vibes — and measure the judge against your own labels before you trust it. An uncalibrated judge produces a stable, reassuring number while quality drops.
• Citations are required and verified. If the model claims a fact, the chunk it came from is shown. "Verified" is the load-bearing word: check that the claim actually appears in the cited chunk, because a model will cite a real document for a sentence that document doesn't support.
• Refusal behavior is defined and tested. The system says "I don't know" when retrieval comes back empty — not hallucinate. Test it deliberately with questions your corpus cannot answer; if those aren't in the eval set, nothing measures this at all.
• Prompt is versioned in code, not hand-edited in a UI somewhere. A prompt is an interface, and so is the JSON shape you promised the model would return — changing either is a forced upgrade on everything downstream that was written against last quarter's output.

Operational

• Latency budgets per stage are measured: embedding, retrieval, rerank, generation. You know which stage is slowest. Optimize time-to-first-token rather than total generation time — a response that starts in 300ms and takes eight seconds reads as fast, while one that appears complete after four seconds of nothing reads as broken.
• Cost per query is tracked. You have a kill switch for runaway prompts. Retrieval that quietly grows from top-5 to top-20 chunks doesn't error; it just multiplies your context bill.
• Logs capture full query/response/retrieved chunks for the last N days, with PII handled. Without the retrieved chunks you cannot tell a retrieval failure from a generation failure after the fact, which is the single most common debugging dead end.
• A/B framework lets you swap retrievers, rerankers, prompts without redeploying code. The corollary is a kill switch: the ability to fall back to the previous prompt, the previous model, or no AI at all, without a deploy.
• An on-call runbook exists for the failure modes you've seen. "Answers went bad" is not an alert someone can act on at 3am; "recall@5 dropped below 0.7 on the rolling eval" is.
• If the corpus is multi-tenant, isolation is enforced inside the query, not by filtering results afterwards — relevance ranks documents, it doesn't check permissions.

Failure modes you'll hit

The ones that bit me hard:

• Document drift. New docs added, old docs removed. Your eval set is stale within weeks. Refresh it monthly, and treat every production failure as a new labelled row — that's the flywheel that makes the set better than the one you originally wrote.
• Query distribution shift. Users start asking about a new topic; retrieval recall drops; nobody notices for two weeks. Catch this with quality alerts on rolling eval scores. Alert on the trend, not on individual scores, or you'll train yourself to ignore it.
• Embedding model upgrades that re-embed everything but break previously-retrieved-fine queries. Always shadow-deploy and compare — and remember every similarity threshold in your code was tuned against a distribution that no longer exists.
• Prompt-injection from retrieved content — a malicious doc says "ignore your instructions and..." Defense: chunk-level allowlisting, output filtering. Retrieved text is data, never instructions, and the boundary has to be enforced somewhere other than the prompt asking nicely.

What I deliberately leave for v2

• Multi-hop reasoning. Most user queries don't need it, and it multiplies both latency and the number of places an answer can go wrong.
• Custom-trained retrievers. Off-the-shelf BGE / OpenAI / Cohere are fine until they're not, and "not" is something your eval set will tell you clearly when it happens.
• Knowledge graph layers. Huge complexity for marginal recall lift on most corpora — worth revisiting only when your failures are genuinely relational rather than lexical.

The pattern in that list: each one is a real technique that solves a problem you probably don't have yet. Adopting them early costs you the ability to tell which change caused what.

If you're shipping RAG, optimize for the boring half first. The boring half is where production lives.

If you're partway down this list and the remaining gaps look familiar, that's the work I do.