LLM observability: what to log, alert on

Most LLM observability advice stops at "log your requests," which is the same advice you'd give a REST API and misses everything specific to this kind of system. An LLM call has cardinality problems a normal request doesn't, alerting thresholds that don't map to error rates, and a replay requirement nothing else in your stack has.

Who this is for: an engineer whose LLM system logs requests and responses the way every other service does, and who's found that those logs don't actually answer "why did this go wrong" when something does.

Cardinality is the problem a normal request log doesn't have

A REST endpoint's logs have bounded cardinality — status code, route, maybe a handful of parameters. An LLM call's meaningful fields are the prompt, the retrieved context, and the generated output, and all three are effectively unbounded free text. Logging them at full fidelity for every request is the default that quietly becomes a storage and cost problem nobody budgeted for, because nobody thought of a log line as a line item.

The fix isn't to log less — it's to log different things at different fidelity, deliberately, rather than uniformly.

What to log unconditionally

• Model and prompt version, pinned exactly — the same pair an agent's audit trail needs, for the same reason: you can't separate a model regression from a prompt regression without both.
• Latency per stage — retrieval, generation, any tool calls — not just total request time. Time to first token, specifically, since that's the number users actually feel.
• Token counts, input and output — this is your cost data before it's your observability data, and the two dashboards should be the same dashboard.
• A pass/fail signal from whatever eval gate the request touched, if it touched one — a production request that happens to look like a labelled eval query is free signal you'd otherwise throw away.

None of these are optional, and none of them are expensive to keep at full fidelity — they're small, structured fields, not free text.

What to sample

Full prompt and response text is the expensive part, and it's also the part you need for debugging a specific incident, not for aggregate monitoring. Sample it — 1-5% of traffic at full fidelity is usually enough to catch a class of problem, with a mechanism to force-capture anything that fails an eval check or trips an alert, so the incident you actually need to debug is never the one that got sampled out.

Retrieved chunks deserve the same treatment as the query and response — without them logged for at least the sampled fraction, a retrieval failure and a generation failure are indistinguishable after the fact, which is the single most common debugging dead end in a RAG system.

What to leave alone

PII in prompts and responses is real and shows up more often than teams expect — names, emails, account details a user typed into a support query. The default should be redaction or hashing before anything hits a log store, not a policy applied after the fact once someone notices. Retention windows matter here too: a log you don't need past 30 days for debugging shouldn't be kept for 180 just because storage is cheap — cheap storage is not the same as free liability.

Alert on drift, not on errors

An LLM system rarely fails with a clean error. It fails by getting quietly worse — recall drifting down, faithfulness scores trending lower, latency creeping up — which means an alerting strategy built around error rates and status codes misses almost everything that actually goes wrong.

Alert on trend against a rolling baseline instead of an absolute threshold: this week's eval scores against last week's, not against a fixed number chosen once and never revisited. A fixed threshold either fires constantly on normal variance or never fires at all once the system has drifted past it gradually enough that no single day looks alarming.

Replay is the requirement nothing else in your stack has

The reason to keep any of this: given a logged request, you should be able to reconstruct exactly what the model saw and re-run it against the pinned model-and-prompt version to get the same output. That's a stronger requirement than most logging exists to satisfy, and it's the one that actually pays off — it's the difference between "the system got worse" and "here is the specific change that made it worse."

If your LLM system's logs currently look like a web server's, closing that gap is usually early work worth doing before scale makes it expensive.