Your AI Agent Has No Colleagues

In the world of AI agents, coordination is not direct but rather relies on the work previously completed to guide the subsequent actions. This is demonstrated by the behavior of various agent frameworks, which exhibit a consistent pattern when encountering failures: create an issue, attempt a retry with validation error, and repeat the process three times with slight modifications to the arguments.

Despite these repeated attempts, the model ultimately issues an apology, which only serves to compound the frustration. The challenge lies in determining whether retrying is a viable solution when presented with a generic error message such as "422 validation error." The model must make an educated guess, as the information provided is insufficient for a definitive decision.

This reliance on guessing, rather than thorough analysis, can lead to inefficiencies and wasted resources. The article suggests refraining from implementing a "don't retry blindly" rule within the system prompt, as it does not address the underlying issue of the model's inability to discern transient failures from permanent ones.

The instinct to log the error and consider it a trail may seem reasonable, but it fails to provide meaningful insights. A more effective approach involves creating a useful failure trace that includes three essential components: a unique identifier for the specific failure, an outcome detailing the next action taken and whether it proved successful, and a denominator to establish the failure rate.

By normalizing the error messages and hashing the resulting string, similar incidents can be identified and counted, allowing for a more accurate assessment of the failure rate. This method eliminates the reliance on a model's guesswork and provides a clear, quantifiable measure of the issue at hand. The author questions whether the failure patterns experienced by different users' agents overlap, suggesting that the shared surface of API calls and MCP servers may be too thin to warrant commonalities in failures.

Further testing is required to determine the extent of overlap in failure patterns. Ultimately, the article prompts readers to reflect on the specific failures their AI agents encounter repeatedly and consider whether these failures are unique to individual users or would be encountered by anyone interacting with the same service.

The key takeaway is that the solution to this problem may lie outside of implementing additional rules or wrappers in the system prompt, and that a thorough examination of the specific failures is necessary to determine the most effective course of action.