LLM Observability: Know What Your Model Did in Production

Shipping an LLM feature without observability is like deploying an API with no logs: it works until it doesn’t, and then nobody can say why. Latency spikes, sudden cost jumps, silent refusals, and hallucinated fields all look the same from a green health check. If the model is part of the product, its behavior has to be visible.

Trace the full request path

A useful trace is more than “model called.” Capture the user action, retrieval steps, tool calls, prompt version, model ID, token counts, latency per stage, and the final decision the product made. When a support ticket says “it invented a refund policy,” you need to replay that path — not guess which dashboard filter might help.

Correlate to product outcomes

Pair model traces with product events: draft accepted, send cancelled, escalation opened, task completed. Observability that only shows tokens and latency tells you the system ran. Observability that shows acceptance and undo rates tells you whether it helped.

Watch the failure modes that matter

Average latency hides the p95 that users feel. Average cost hides one tenant burning tokens on retries. Classify failures into buckets you can act on: timeouts, empty retrieval, schema validation misses, policy refusals, and low-confidence escalations. Alert on rate changes in those buckets, not on a single vague “AI error” counter.

• Log prompt version and model ID on every generation
• Record token usage and estimated cost per request and per tenant
• Store retrieval hit/miss and citation presence when answers are grounded
• Track accept, edit, undo, and escalate rates next to raw model metrics
• Sample full traces for review — redact secrets before they land in storage

Close the loop with evals

Production traces feed better eval sets. When a real failure escapes, turn it into a regression case before you change the prompt or the retrieval config. Offline scores without production feedback drift into theater; production metrics without evals leave you reacting forever.

“If you cannot answer which prompt version, which retrieved docs, and which cost bucket produced a bad answer, you are not operating an AI product — you are hoping.”

Make ownership explicit

Someone on the team should own the dashboards, the alert thresholds, and the weekly review of sampled failures. At Brutanix Studios, we treat LLM observability as part of the shipping surface — same as API monitoring — because “the model acted weird” is not a root cause.