Best Practices for Monitoring Deployed AI Agents

Best Practices for Monitoring Deployed AI Agents: Correct Behavior, Reliability, Cost, and Safety

Monitoring deployed AI agents requires more than just checking HTTP status codes and latency. An agent's failure is not limited to throwing exceptions; it can happen by making plausible decisions badly, like using the right tool with wrong arguments or relying on stale data. This guide insists on four crucial dimensions of monitoring: behaviour, reliability, cost, and safety. Focusing on just one dimension leaves you blind to potential problems.

Without capturing a trace for each run, there is no basis for aggregation, alerts, or debugging. OpenTelemetry's gen_ai conventions provide a standard shape for traces, including token counts and a run ID to tie spans, metrics, and evaluations together. This vendor-neutral approach ensures the instrumentation remains consistent, even if the backend changes.

Sensitive data, such as prompts containing confidential information, must be redacted before being stored. Otherwise, it poses a leak waiting for an audit. Instead of monitoring totals, focus on ratios like total spend per successful outcome, step count against its ceiling, and tool-call error and retry rates. The most critical metric is the disagreement rate between the agent's self-reported success and an independent outcome check, which helps detect silent partial successes.

A dashboard displaying task success rate, latency, token cost, tool success, and step count serves as a starting point but not a final solution. Golden sets of representative tasks, re-run on every model, prompt, or tool change, are vital to detect changes that pass unit tests but degrade real work. Live scoring and treating falling eval scores as deploy-blocking signals ensure that the monitoring process remains effective.

A calibrated LLM judge is essential to measure the quality of agent outputs. In Zheng and colleagues' 2023 study, GPT-4 achieved an 80% agreement rate with humans, comparable to two humans agreeing on each other's assessments. However, this agreement often masks failures and carries known biases toward longer answers and the model family. Regular calibration is necessary to maintain the reliability of the judge.

Loops and prompt injection are not edge cases but recurring production issues. Implementing deterministic circuit breakers at the orchestrator level, with hard step caps, per-tool retry counters, and cost caps enforced in code, is crucial to address these problems. Security measures, such as monitoring patterns in the trace stream, are essential to detect prompt injection, excessive agency, and improper output handling.

Alerting should be based on rates over a rolling window for events that can wait, while sharper alerts should trigger immediately for safety violations, prompt-injection attempts, or runs exceeding their cost caps. Every alert must have a runbook with the trace attached, and any production failure that reaches a human should become a structured case in the golden set, ensuring the monitoring stack evolves and improves over time.