Agent Oversight Needs Metrics, Not Just Logs

Anthropic has shared metrics on its research practices, focusing on three main areas: the proportion of AI-led research, oversight of internal agents, and allocation of compute resources. The key takeaway for software teams is not merely whether their systems mimic a frontier lab, but rather whether their agent operations can provide concrete answers to specific questions.

For instance, what actions did the agent control? Which actions were reviewed? How long did the review process take? What evidence supports the conclusions? Anthropic's data, published in September 2026, offers concrete examples of internal systems and time frames. However, these figures are not industry-wide averages and should not be treated as such.

Instead of merely tracking agent activity, teams should concentrate on delegated work. Anthropic describes a six-level automation scale, ranging from no AI involvement to fully autonomous work. It's crucial to differentiate between a model completing most of a task with supervision versus having the authority to deploy every result.

Recording task details, such as type, delegation level, accountable individual, allowed actions, and final approval point, provides a more comprehensive picture than just looking at session counts, tokens, or tool calls. Anthropic's oversight metrics include coverage, review latency, and escalation rate. Coverage refers to the share of actions sent through monitoring, review latency is the time between an action and review, and escalation rate is the share of actions blocked, redirected, or flagged.

These metrics, when considered together, offer more insight than a single flag status. In their most widely used internal platform, Anthropic reports that roughly 30,000 agents were active simultaneously in August 2026. Online monitoring blocked just 0.002% of the over a billion decisions analyzed that month, while the offline process escalated about 50 high-priority transcripts per week to human review.

While these figures illustrate the monitoring process on that platform and period, they do not reflect the overall rate of harmful agent behavior or the monitors' detection accuracy. To create a robust reporting system, teams should record key details for each consequential action. This should include a stable agent identity, task ID, and action ID, along with the action's target and risk class.

It's essential to avoid exposing sensitive information in logs. The decision-making process should also record timestamps for when the action occurred, when review took place, and when a person resolved any escalations, along with the policy version and evidence used. Monitoring coverage can be calculated by dividing monitored actions by eligible actions, segmented by action class.

Latency should be measured separately for automated and human review, and escalations must include outcomes, as a high flag count could indicate a sensitive monitor, a noisy one, or a genuinely risky workload. For operations with immediate, irreversible harm potential, a review must occur within a reasonable timeframe. While retrospective analysis is valuable for identifying patterns and improving controls, delays in addressing slower issues must be clearly documented.

Finally, Anthropic reports that about 6% of compute allocated to AI R&D went to safety work during a specific week in July 2026, compared to about 12% for AI-driven AI R&D. These figures are estimates based on Anthropic's definitions and usage categories. It's crucial to publish the categories and classification rules for transparency.

Ambiguous work should be clearly marked to prevent false precision. An operational contract for agent platforms should include questions about delegated work, evidence to retain, what the agent could do, permissions and approval boundaries, whether actions were checked, monitor decisions, timestamps for reviews and resolutions, and resource allocation details.

This framework, though not a guarantee of safety, provides testable standards for evaluating agent operations.