Article
Traditional monitoring answers "did the request succeed and how long did it take," and an LLM call can succeed, return in 400ms, and still be completely wrong. That gap is why LLM observability is a distinct discipline rather than a dashboard bolted onto existing APM: the failure mode that matters most, a confidently wrong answer, produces no error, no exception, and no slow trace.
Log the full input, not a summary of it
Every request needs the exact prompt sent to the model, not a paraphrase of it — the system instructions, the retrieved context if any, the conversation history, and the final assembled prompt as the model actually received it. When an answer is wrong, the first question is always "what did the model actually see," and a system that only logged the user's original question cannot answer it, because everything assembled between the question and the model call is where the bug usually is.
Log the output and the decision path
The raw output, any tool calls made and their results, and — for a RAG system — exactly which chunks were retrieved and which ones the model actually cited. For an agent, every step of the loop: the action proposed, the tool result, and the next decision, not just the final outcome. A log that only captures the final answer cannot distinguish "retrieval found nothing relevant" from "retrieval found the right thing and the model ignored it" — two completely different bugs with the same symptom.
Cost and latency, per request, per feature, per tenant
Aggregate cost dashboards hide which specific feature or tenant is driving spend, which is exactly the blind spot that let one tenant's usage inflate an entire platform's bill before anyone noticed. Per-request granularity, rolled up by feature and tenant, turns "the bill went up" into "this feature's average token count doubled last Tuesday," which is an actionable finding instead of a mystery.
Quality signals, not just technical ones
A request that returns 200 OK in 300ms with a wrong answer looks identical to a correct one in a standard dashboard. Quality signals — user thumbs-down, a follow-up question that suggests the first answer failed, a flag from the evaluation set running in production on a sample of live traffic — need to be first-class metrics next to latency and error rate, or the team monitoring the system will have no idea it is degrading until someone complains.
Trace a bad answer back to its cause
The point of all this logging is answering one question fast: given a specific wrong answer a user reported, what happened. Good observability lets you pull the exact prompt, the exact retrieved chunks, the exact model version, and the exact output for that one request, in one place, in minutes. Without it, debugging a reported bad answer means asking the user to reproduce it and hoping the same wrong retrieval happens twice.
Feed it back into the evaluation set
The most valuable output of an observability system is not the dashboard; it is the stream of real failures it surfaces, which become new cases in the evaluation set that prevent the same failure from shipping again. Observability and evaluation are not two separate practices — one finds what's broken in production, the other proves the fix, and a fix without a new test case is a fix that can silently regress on the next model upgrade.
Where this fits architecturally
Observability is not a feature added to one part of an LLM application; it needs to be wired through every layer — model, harness, data, and product surface — emitting into one place with one schema, so a problem anywhere in the stack is diagnosable with the same tools. It is a standing requirement in every AI/ML Development system we ship, built in from the first request rather than added after the first unexplained complaint.