Choosing a model in 2026: not a benchmark
A decision tree based on task shape — latency, accuracy, cost, context length — beats chasing leaderboard rankings your actual queries don't match.
A leaderboard ranks models on someone else's benchmark, and the same problem that breaks an embedding-model swap breaks a leaderboard-driven model choice: a model that wins by three points on average is not three points better on every query, and public benchmarks aren't built from your queries. The decision that actually holds up is based on the shape of the task, not the leaderboard position.
Who this is for: an engineer about to pick a model for a new feature by checking which one is on top of a benchmark this month, instead of asking what the task actually needs.
Why the leaderboard doesn't transfer
A benchmark score is an average across a fixed set of tasks that almost certainly don't match your query distribution — the same gap that makes a vendor's own performance numbers not transfer directly to your app shows up here for the same underlying reason: a number measured on someone else's workload tells you about their workload first and yours only by coincidence.
The practical failure mode: a model wins a coding benchmark and gets chosen for a customer-support classifier, because "it's the best model" got treated as a property of the model rather than a property of the task it was measured on.
The decision tree
Four questions, asked in this order, because each one eliminates more of the field than the next:
1. What's the latency budget? Time to first token is what users feel, not total generation time. A sub-second interactive response and a batch job that runs overnight aren't choosing from the same model tier — the interactive case needs a smaller, faster model regardless of what else is true, because no accuracy gain survives a response that reads as frozen.
2. How much does an error cost? A drafting assistant that a human reviews before anything ships tolerates a wrong answer that a compliance-adjacent classifier can't. This is where accuracy requirement gets set — not as a vague "as accurate as possible," but as a specific answer to "what happens when this is wrong," which tells you how much you can trade accuracy for speed and cost in the first place.
3. How much context does the task actually need? Not how much a model supports — how much the task needs. A single-document Q&A task and a task that reasons across fifty retrieved chunks have different requirements regardless of what context window either model advertises, and paying for a window you don't fill is paying for capability the task can't use.
4. What does it cost at your actual volume? The cost dashboard from the previous post is what this question is really asking — not "what's the per-token price" but "what does this model cost per month at the volume this feature will actually see," which is a different number for a feature serving ten requests a day than one serving ten thousand.
The tree outputs a tier, not a name
Deliberately: "a fast, cheap model for high-volume low-stakes classification" is a stable answer. "GPT-4o-mini" or whatever the current instance of that tier is called is not — model names and specific releases change every few months, and a decision tree built around today's model name is stale by the time anyone reads it again. The tier is what to commit to in a design doc. The specific model is what to pin in config, re-evaluated whenever a new release changes the tier's field.
Where routing fits
Most systems don't need one answer to this question — they need several, applied per-request. Route on the request's own difficulty, not on a single upfront choice for every call the feature makes. The decision tree above is how you set the tiers a router chooses between; it's not an argument for picking one model and using it for everything.
Verify on your own eval set, not the leaderboard
Whatever tier the tree points to, the same rule as everywhere else in this cluster applies: confirm it on your own eval set before trusting it in production, because a benchmark ranking and your actual query distribution are two different things, and only one of them is the one your users experience.
If you're choosing a model for a new feature and the current plan is "whichever one's on top this month," working through task shape first usually changes the answer.