Uptime Is Not an Agent SLO

An agent endpoint returns an HTTP 200 status code for the majority of requests, indicating that the service is functioning. However, not all successful responses involve retrieving the intended data, some tool calls exceed their budget limits, and a few write actions take place without the necessary approval. The service appears to be available, but the product may not be performing as expected.

Traditional service-level indicators (SLIs) are still relevant, but agent systems introduce an additional layer of complexity. The path and outcome can fail even if the request handler itself does not fail. To address this, user-facing promises should be broken down into distinct indicator families. First, the outcome: did the workflow produce the intended externally observable result?

This can be measured by the task-success rate, which is the ratio of eligible tasks with a verified outcome to the total number of eligible tasks with a known outcome. It is crucial to exclude cancelled requests or unsupported tasks from the calculation, following a documented policy. Second, safety and policy: did protected actions meet the required controls?

The authorized-action rate measures the ratio of protected actions with valid approval to all protected action attempts. Forbidden actions should also be tracked separately. Third, efficiency: did the useful result arrive within a bounded resource envelope? Indicators include task latency, model calls per task, total tokens, tool retries, and cost per verified outcome.

The focus should be on optimizing for task success rather than minimizing token usage. Fourth, autonomy quality: did the agent resolve work without unnecessary escalation or excessive intervention? An escalation is not always a failure, especially for ambiguous or high-risk tasks. In such cases, seeking human intervention might be the appropriate behavior.

Finally, user experience: did the agent resolve work without unnecessary escalation or excessive intervention? A single "agent reliability score" that blends multiple indicators can obscure important tradeoffs. Instead, separate objectives should be defined, each with its own SLI. For example, availability measures valid responses versus requests, task outcome tracks verified successes versus eligible tasks, safety measures controlled actions versus protected attempts, efficiency considers latency or calls per outcome, and escalation assesses appropriate escalations versus eligible cases.

Targets for these objectives should be derived from product risk, user expectations, and observed baselines. Violations should be handled as separate engineering work items, with clear definitions of when budgets are exhausted. While uptime is necessary, it is not sufficient to determine whether an agent system is performing as intended.