I Ran 4,200 Trials Testing LLM Agent Reliability. Here’s What Broke.

In the course of testing an AI agent's reliability, I conducted 4,200 trials designed to uncover potential problems. ReliAgent, the reliability assessment tool I created, identified signals indicating when responses from tool calls should be treated with caution. To rigorously test ReliAgent, I developed Basanos, a validation program that subjected it to adversarial trials in various experiments.

Through Basanos-2 and Basanos-3, I ran a total of 4,200 trials, carefully examining the results for both successful and failed scenarios. The benchmark revealed genuine product issues that were addressed prior to publication. However, during the process, I encountered a valuable lesson about interpreting benchmark results. One experiment involving a sycophantic behavior exhibited inconsistent detection rates across different models, which initially appeared as a detector failure.

Upon closer inspection, it became clear that Claude Haiku 4.5 did not exhibit this behavior in the first place, but rather, it pushed back against the premise presented to it.

This distinction is crucial when assessing AI systems that detect model behavior. Two key questions emerge: Did the model generate the failure condition? And did the detector recognize it? Treating these questions as the same could lead to misleading benchmark outcomes.

Additionally, I discovered that language-dependent detectors showed varying performance across different models, while metadata-driven detectors evaluated in Basanos-3 consistently achieved perfect true positive rates and false positive rates across all tested model families. Furthermore, cross-provider testing unveiled output differences and a configuration dependency that must be considered in real-world deployments.

Through this extensive trial process, I learned several key principles to apply in future benchmarking efforts. Firstly, testing mechanisms, not just metrics, is essential. Secondly, understanding the specific conditions created during testing is vital to accurately determining whether a detector identified a genuine issue. Thirdly, maintaining clean controls is crucial, as a system that flags everything isn't useful.

Lastly, separating model behavior from detector behavior is paramount - a model failing to exhibit an expected failure mode isn't necessarily a detector's false negative.

Testing across multiple model families proved to be invaluable in this process. A single-model benchmark only provides insight into the specific environment, rather than generalizing to other models. Lastly, I learned the importance of keeping mistakes. Documenting superseded experiments and corrective runs preserves the research record and allows for continuous improvement.

Ultimately, I realized that designing benchmarks willing to expose flaws in your product is a crucial aspect of proper testing. If the only acceptable outcome is proving your product's success, you're not truly evaluating its effectiveness. In the future, I will continue sharing insights from these experiments as I further develop ReliAgent and Basanos through HDGForge.