Monitor Quality, Drift, Latency, and Cost

Monitor the service and the decision

Infrastructure signals include availability, errors, queue depth, p50 and p95 latency, saturation, token use, and cost. Data signals include schema violations, missingness, range changes, category frequency, embedding shifts, and source freshness. Product signals include completion, correction, escalation, abandonment, complaints, and downstream outcomes.

Quality labels may arrive late. Define immediate proxies and a process for joining delayed outcomes back to the exact model, prompt, retrieval index, and policy version that produced a decision.

Interpret drift carefully

Input drift means the observed distribution changed; it does not prove the model became worse. Concept drift means the relationship between inputs and outcomes changed. Investigate by segment, data source, time, and version. Compare with seasonality and known product changes.

For LLM systems, monitor retrieval failures, unsupported claims, refusal behavior, tool errors, policy events, citation validity, grader disagreement, prompt changes, model-provider changes, and context length.

Make alerts actionable

Every alert needs threshold rationale, severity, owner, runbook, dashboard link, evidence query, escalation path, and recovery action. Test that the alert fires and that on-call staff can identify the affected version and users.

Practice: production scorecard

Design a dashboard with four service, four data, four quality, and four product signals. For each, state aggregation, segment, threshold, owner, and response. Simulate one latency incident, one data shift, and one quality regression.

Close the loop

Monitoring is complete only when incidents become evaluation cases, fixes are regression-tested, affected artifacts are traceable, and the team records whether the response reduced real user harm.