Your worst agent bugs never get reported

When running AI agents in production, issues often go unnoticed until they impact users. Logging traces and manually searching for problems is slow and inefficient, failing to identify many critical bugs. To address this, Opik developed Diagnostics, an automated agent debugging tool that actively identifies potential issues in agent behavior.

The development process involved several iterations, starting from a daily report feature that summarized overall activity patterns. The team built a sub-agent called TraceInvestigationAgent to investigate suspicious traces, but it hit scaling limitations. They then experimented with splitting the debugging work between two models, but this was too costly for high-volume use cases.

The breakthrough came when they realized they could leverage ClickHouse, a column-oriented database, to query trace data directly. By providing the main agent with a tool to run SQL queries, it could efficiently analyze trace prevalence and prevalence patterns without reading individual traces. This change not only improved the accuracy of identified issues but also reduced costs and improved scalability.