Cost-aware LLM engineering
Prompt caching, model routing, and compression actually move an LLM bill. What each one does, what it doesn't, and the dashboard to build before you need it.
Nobody searches for LLM cost optimization until the bill spikes, and by then the fix is usually a scramble instead of an architecture. The four levers below are the ones that actually move a bill, in the order I'd apply them — and the dashboard that tells you which one to pull, which most teams build only after the first spike.
Who this is for: an engineer whose LLM spend just became a line item someone asks about, and who's about to guess at a fix rather than measure where the money is going.
Prompt caching: what actually caches
Anthropic and OpenAI both discount tokens that match a previous request's prefix — Anthropic at roughly a 90% discount on cache hits, OpenAI similarly for its own automatic prompt caching. The catch is the word prefix: caching only helps the portion of the prompt that's byte-identical and appears first.
That means system prompts, tool definitions, and few-shot examples cache well — they're stable and they lead the request. Retrieved context and conversation history cache badly by construction, because they change on every call. The fix isn't clever cache logic; it's ordering: put everything stable first, everything variable last, so the stable prefix is as long as it can possibly be. Reordering a prompt for this reason alone is usually the single cheapest win available, because it costs an afternoon and nothing else about the system has to change.
Model routing: send easy queries to a small model
Not every request needs your best model. Which model, for which tier of request, is its own decision — this section assumes you've made it and covers the routing on top. A classifier — or often just a confidence threshold on the small model's own output — can route simple queries to something in the Haiku or Mini class and escalate only the ones that need it.
The trap is routing on query type instead of query difficulty. "Is this a support question" is the wrong classifier; "does this need multi-step reasoning" is closer. A short query can be genuinely hard, and a long one can be trivial. Route on the signal that predicts whether the small model will actually get it right, and log every escalation — the escalation rate is your signal for whether the router is too conservative or too aggressive, and it drifts as your traffic mix changes.
Prompt compression: shrink the context, not the prompt
Techniques like LLMLingua and summary-based memory reduce the tokens you send without (ideally) losing what the model needs. This is real, but it's the smallest lever of the four, and it's the one most likely to quietly cost you quality — compression is lossy by definition, and the loss doesn't show up as an error, it shows up as a slightly worse answer that's hard to attribute to anything.
Apply it last, and only to the part of the prompt that's genuinely too long to fit — retrieved context on a wide corpus, long conversation history — not to the system prompt or instructions, where every word is there because it was needed once.
When to fine-tune
Rarely, and the threshold is volume plus stability: you need a task with high enough call volume that the training cost amortizes, and a task stable enough that the training data won't be stale in a month. Below that threshold, a well-cached, well-routed prompt beats a fine-tuned model on cost, because you're not paying to retrain every time the task shifts slightly — and in most production LLM systems, the task shifts more often than teams expect. The same math applies to an embedding swap: the model call is on your critical path either way, and the decision to change it should be made on evidence, not on a leaderboard.
You don't need a case study to know which lever to pull first — the dashboard below tells you, on your own numbers, within a day of building it. The order in this post (caching, then routing, then compression, then fine-tuning) is the order of effort-to-savings ratio, not a ranking that requires your specific bill to be true.
The dashboard you should build day one
One table, one row per request: model, input tokens, output tokens, cache hit or miss, cost, and the feature or endpoint that triggered it. That last column is the one teams skip and the one that matters most — total spend tells you the number is too high, but only spend-by-feature tells you which feature to fix, and without it every optimization decision is a guess dressed up as data.
Cache hit rate deserves its own line on that dashboard, tracked over time the same way retrieval recall gets tracked rather than measured once — a hit rate that degrades silently as your prompts drift is exactly the kind of regression nothing else will surface.
If your LLM spend is a line item someone's now asking about and you don't have this table yet, building the visibility before the optimization is usually where I'd start.