85% of companies burned by an AI mistake are racing to cut the humans who might catch the next one

In a recent study, 85% of companies that experienced an AI agent failing in production are rapidly removing human oversight from deployment decisions, despite a growing trust in automated evaluation. In July, 13% of 108 surveyed enterprises expressed trust in automated evaluation, up from just 5% in the previous month. However, the survey also shows that poor alignment between tests and real-world results, companies' biggest concern, dropped by 10 points, from 29% to 19%.

Despite this increase in trust, 49% of respondents reported that an AI agent or LLM-powered feature that cleared company testing later caused a customer-facing problem, a figure that remained unchanged from June. Additionally, nearly a quarter (24%) of respondents said this issue had occurred more than once. The research reveals that the gap between confidence in automated evaluation and the effectiveness of preventing failures is growing.

Of the enterprises that experienced a test-passing agent failing in the real world, only 4% placed complete trust in automated checks, while 24% of those with no comparable incident expressed full confidence in the automated process. Companies struggling with this issue, such as automated agent error monitoring and mitigation platform Raindrop.ai, are witnessing a shift in the market, with fewer resources dedicated to evaluation and more focus on anomaly and issue detection solutions.

The findings are based on a survey of 108 enterprises with at least 100 employees, predominantly midsize organizations. The data suggests that companies that have faced AI failures are more likely to adopt a no-approval model for deployment automation, compared to those without such incidents.