LLM Observability Runbook: Diagnose Hallucinations, Latency, Errors, and Quality Drift
When an LLM application becomes inaccurate, slow, expensive, or unreliable, observability turns a vague complaint into a traceable engineering problem. This runbook explains what to record, how to establish useful baselines, how to investigate common symptoms, and when to revisit your monitoring after model, prompt, traffic, or retrieval changes.
Overview
LLM observability is the practice of collecting enough application, model, retrieval, and user-outcome data to explain what happened during an AI interaction. Traditional application monitoring can tell you that a request returned an error or exceeded a latency threshold. An LLM application also needs evidence about the prompt, retrieved context, model configuration, generated response, tool calls, token usage, and evaluation result.
The goal is not to log everything indefinitely. The goal is to make important failures reproducible while protecting sensitive data and keeping telemetry useful. A good observability design helps answer five questions:
- What did the user ask, and what instructions and context reached the model?
- Which model, application version, prompt version, and retrieval configuration handled the request?
- Where did time and cost accumulate?
- Was the response factually supported, correctly formatted, safe, and useful?
- Is this an isolated failure, a new trend, or a change affecting a particular route, tenant, model, or document set?
Apply this runbook after an incident, during weekly quality reviews, and whenever you change a model, system prompt, retrieval pipeline, tool schema, output parser, or traffic pattern. It complements a prompt testing and regression framework; production traces show how the system behaves with real inputs, while pre-release tests help catch known failures before deployment.
What to track
1. Use a trace with meaningful spans
Represent each user request as a trace with related spans. A typical trace might contain an API request, authentication, input moderation, retrieval, prompt assembly, model generation, tool execution, output validation, and response delivery. This structure lets you distinguish a slow vector search from a slow model call, or a model error from a failure in your own parser.
At minimum, record these trace fields:
- Request identity: trace ID, request ID, timestamp, environment, region if relevant, route, tenant or workspace identifier, and correlation ID.
- Version information: application release, model identifier, provider or endpoint, system prompt version, template version, retrieval index version, tool definitions version, and feature flags.
- Input characteristics: task type, language, modality, approximate input size, attachment type, and whether the request was a retry. Avoid storing raw personal or confidential content unless there is a defined reason and retention policy.
- Generation settings: maximum output limit, sampling settings, stop conditions, structured-output mode, and timeout configuration.
- Retrieval details: query representation, index or collection, number of candidates, number of final chunks, scores or ranking signals, filters, document identifiers, and retrieval duration.
- Tool activity: tool name, validated arguments, start and end times, result status, retry count, and a redacted result summary.
- Outcome: status code, finish reason, output length, parser result, citation or grounding checks, user feedback, and downstream business outcome where available.
Keep raw prompts and responses separate from operational metadata when possible. Redact secrets, credentials, payment data, personal information, and proprietary content before sending telemetry to a logging or tracing system. Define access controls and retention periods with the same care used for other production data.
2. Measure reliability, quality, speed, and cost separately
A single success rate hides too much. Track a small set of clearly defined metrics:
- Availability: proportion of requests that receive a valid application response rather than a timeout, provider error, network error, or internal failure.
- Valid response rate: proportion of responses that pass schema, parsing, and required-field checks.
- Grounding or support rate: proportion of sampled answers whose important claims are supported by supplied documents or approved data.
- Task success: proportion of responses judged to complete the user’s intended task, using human review, deterministic checks, or a suitable evaluator.
- Latency: measure time to first token and total response time separately. Report percentiles such as median and a high percentile rather than only an average.
- Token and cost usage: input tokens, output tokens, cached or reused context where applicable, retries, and estimated request cost using your current accounting method.
- Feedback signals: regeneration rate, correction rate, abandonment, escalation to a human, thumbs-up or thumbs-down, and support reports.
Define each metric before putting it on a dashboard. For example, decide whether a timed-out request counts as a failed request, whether retries are counted once or individually, and whether evaluator scores are calculated per conversation or per turn.
3. Use a symptom-to-cause diagnostic table
| Symptom | Likely causes to check | First diagnostic action |
|---|---|---|
| Answers are inaccurate or unsupported | Retrieval returned weak context; prompt or model changed; source content is stale; context was truncated; tool result was misread. | Compare the failing trace with a known-good trace. Inspect retrieved document IDs, final prompt sections, context length, and grounding checks. |
| Answers are slow | Large inputs or outputs; retrieval or tool latency; provider queueing; retries; serial tool calls; overloaded application workers. | Break total latency into span durations and compare time to first token with generation time. |
| Requests fail intermittently | Timeouts; rate limits; transient provider errors; malformed tool arguments; connection limits; parser failures. | Group errors by status, provider response, route, model, region, and retry number. Check whether retries improve or worsen the failure rate. |
| Costs rise without more users | Longer prompts; duplicated history; retrieval returning more text; verbose outputs; retry loops; fallback routing changes. | Compare input and output tokens per request against the baseline, split by route and prompt version. |
| Quality declines gradually | Source documents changed; user mix shifted; model behaviour changed; prompt edits; evaluator drift; new edge cases. | Review a consistent sample over time and segment scores by task, language, tenant, model, and document collection. |
| Structured output breaks | Schema change; unsupported fields; truncation; prompt ambiguity; escaping or encoding issue; parser mismatch. | Store validation errors and inspect the raw model output in a protected debugging view. |
Cadence and checkpoints
Observability becomes more valuable when checks happen on a predictable schedule. Use the following cadence as a starting point and adjust it to your risk, traffic, and release frequency.
Per request
Capture the trace, status, latency spans, model and prompt versions, token counts, retrieval and tool activity, validation result, and a privacy-safe outcome classification. Alert on service-level failures that require immediate action, such as sustained error rates, broken authentication, or a queue that is preventing requests from completing.
Daily or weekly
Review a sample of successful and failed traces. Include low-feedback requests, regenerated answers, unusually long requests, parser failures, and requests near token or timeout limits. Examine quality by route rather than relying on one global score. A customer-support summarizer, document question-answering flow, and coding assistant may have completely different failure patterns.
After every material change
Record a before-and-after comparison when changing the model, provider, system prompt, retrieval settings, chunking strategy, tool definitions, safety rules, output schema, or fallback route. Run a fixed evaluation set and compare production samples after release. The comparison should include quality, valid response rate, high-percentile latency, token usage, error categories, and user feedback.
Monthly or quarterly
Refresh dashboards, alert thresholds, trace sampling rules, redaction rules, and runbook ownership. Recheck provider status and telemetry capabilities before depending on a new field or feature. Review whether the model identifier, API response shape, pricing inputs, usage counters, and tracing integrations still match your implementation. This maintenance is especially important when APIs and model families evolve.
A practical baseline template is:
- Period: [date range]
- Traffic: [requests, active users, routes]
- Quality: [task success, grounding rate, valid response rate]
- Reliability: [availability, timeout rate, parser error rate]
- Latency: [median and high-percentile time to first token and total time]
- Usage: [input tokens, output tokens, retries, average tokens per request]
- Top failure categories: [ranked list with counts and affected versions]
- Comparison: [previous period, release baseline, or evaluation baseline]
How to interpret changes
Start with segmentation, not assumptions. A global quality decline may come from one route, one customer workflow, one language, or one retrieval collection. Filter by application version, model, prompt version, user intent, document set, input size, and tool usage before changing the prompt or model.
For hallucination debugging, inspect the full chain in this order:
- Was the user request classified correctly?
- Did retrieval return relevant and sufficiently current material?
- Did prompt assembly preserve the instructions and context, or did truncation remove important content?
- Did the model follow the expected answer policy and format?
- Did a tool return an error, partial result, or ambiguous value?
- Did post-processing alter, omit, or incorrectly display the answer?
Do not treat every unsupported answer as a model-only problem. In a retrieval-augmented application, poor chunking, incorrect filters, stale indexes, duplicate context, and missing citations can be equally important. In a tool-using application, validate arguments before execution and record tool outcomes independently from the model’s final prose. For security-sensitive workflows, pair operational monitoring with the prompt injection prevention checklist.
For latency, calculate a budget for each span. If retrieval consumes most of the request time, changing the model may not help. If time to first token is acceptable but total time is high, inspect output length, stop conditions, and streaming behaviour. If both are high, investigate input size, provider response time, queueing, and retries. For cost, compare tokens per successful task rather than cost per raw request; repeated failures and regenerations can make a seemingly cheap route inefficient.
When a baseline moves, first establish whether the change is real. Check deployment timing, traffic mix, dashboard query changes, missing telemetry, evaluator version, and sampling bias. Then reproduce a small set of representative traces. Make one controlled change at a time where practical, and keep the original trace, configuration, and evaluation result attached to the investigation.
Sampling guidance
Full-fidelity storage may be unnecessary for every request. Keep complete traces for errors, timeouts, schema failures, policy-triggered cases, and a small representative sample of successful requests. Sample by route and traffic segment so high-volume paths do not hide low-volume critical workflows. Increase sampling temporarily during an incident or after a release, then return to the normal policy after confirming stability. Always preserve enough metadata to explain why a trace was sampled and how sensitive content was handled.
When to revisit
Revisit this runbook after any incident involving incorrect answers, elevated latency, unexpected spend, provider errors, tool failures, or output-format regressions. Also schedule a recurring review at least monthly for active systems and a deeper quarterly maintenance review for telemetry, dashboards, alert thresholds, sampling, retention, and provider integrations.
Use this escalation workflow:
- Stabilise: confirm the impact, protect users, pause a risky rollout, enable a known-good fallback, or reduce traffic to the failing route.
- Scope: identify affected versions, models, tenants, task types, regions, and time windows. Separate confirmed failures from missing telemetry.
- Trace: inspect representative good and bad requests across retrieval, generation, tools, validation, and delivery.
- Mitigate: apply the smallest reversible change, such as reverting a prompt, reducing context, disabling a failing tool, or adjusting a timeout.
- Verify: compare the same quality, reliability, latency, and usage metrics against the baseline before closing the incident.
- Learn: update tests, dashboards, alerts, documentation, ownership, and sampling rules.
Capture the postmortem in a consistent format:
- Incident: what failed and when
- Impact: affected users, routes, tasks, and duration
- Detection: which signal identified the problem
- Timeline: releases, alerts, investigation steps, and mitigations
- Root cause: technical cause and contributing conditions
- Evidence: trace IDs, metric comparisons, and representative examples
- Resolution: what restored acceptable behaviour
- Prevention: tests, alerts, safeguards, or process changes
- Owner and due date: who will complete each follow-up action
Keep the runbook beside your deployment and review process. Pair it with the LLM cost optimisation guide for usage investigations, the LLM API migration guide before provider changes, and the AI model selection guide when a quality or latency problem may require a different model. The most useful observability system is not the one with the most charts; it is the one that lets an engineer move from symptom to evidence, mitigation, and a measurable follow-up.