When should an agent stop?

When it comes to determining when an agent loop should cease its iterations, there are several key factors to consider. Traditionally, there are two primary methods: setting a fixed maximum number of iterations or allowing the agent to self-declare when to stop. A recent benchmark conducted by CDV, an open-source judge for coding agents, demonstrated that a Bayesian adaptive stop rule utilizes 41% fewer steps than a fixed six-step budget while still achieving a 99.7% success rate on tasks meeting a quality threshold.

Upon replicating these findings, three significant observations emerged. Firstly, the adaptive policy essentially incorporates a plain threshold rule (stopping at the first score equal to or greater than 0.80) as its foundation. This occurs due to the order in which the guards are evaluated within the policy stack. Secondly, both the adaptive policy and the threshold rule falter when faced with noisy judge scores.

At a per-step noise level of σ = 0.10, the loop mistakenly believed it had reached the quality bar in 99.5% of runs, whereas the actual success rate was only 71%. Remarkably, a fixed budget of six iterations consistently maintained a true reach of 93% across all noise levels, significantly outperforming the adaptive approach in noisy conditions.

Lastly, introducing a simple two-step rule that requires consecutive scores above the bar to be confirmed restored true reach to 97% at a noise level of 0.10, albeit requiring 1.4 additional steps compared to the adaptive policy.

The cornerstone of this issue lies within the agent loop itself. An agent loop operates through a repeat-until mechanism, where it acts, evaluates the outcome, and then decides whether to continue based on a set of predetermined guards. The decision-making process is where complications arise. A fixed maximum iteration count either proves insufficient for challenging tasks or wasteful for simpler ones.

Conversely, empowering the agent to assess its own performance means that the entity responsible for deciding when to stop is the very same entity being judged. This creates a conflict of interest, as agents tend to optimize reported progress rather than genuinely determining when to halt.

To address these challenges, CDV's solution was to decouple the score evaluation from the decision-making process. At each step, the agent provides a score, which is then scored by a judge. A policy stack, evaluating guards in a specific order, determines whether to continue. The policy includes guards for score thresholds, plateaus, Bayesian estimates of expected improvement, and other metrics such as a maximum budget, wall-clock time, token limits, and repeated outputs.

This layered approach allows for more nuanced and reliable stopping decisions, particularly in scenarios where the judge's score might be unreliable due to noise.

The benchmark analyzed four strategies across a range of synthetic tasks. The results revealed that the adaptive policy's enhancements over the simplest threshold rule are minimal and not easily discernible through statistical analysis. While the adaptive approach can stop earlier than the threshold rule through the use of plateau or Bayesian guards, it does not stop later than the threshold rule.

This finding highlights that the added complexity of the adaptive policy does not significantly benefit the overall stopping decision process in this particular benchmark.

Additionally, the benchmark explored the impact of noisy judge scores on the decision-making process. When faced with such noisy inputs, the threshold rule consistently reaches the quality bar in nearly all runs, but the true success rate drops drastically as noise levels increase. The adaptive policy, similarly, is affected by noisy scores, leading to premature decisions to stop.

In contrast, a fixed six-step budget maintains a stable true reach across varying noise levels, emphasizing the importance of robust decision-making policies that can handle uncertainty effectively.

In summary, the key takeaways from this replication and extension of the CDV benchmark are clear: the adaptive policy's advantages over a straightforward threshold rule are limited, particularly when the judge's scores are noisy. Trusting a single score at face value can lead to suboptimal stopping decisions, especially in noisy environments.

Implementing a simple two-step confirmation rule can help mitigate the effects of noise and improve the reliability of stopping decisions. Ultimately, the decision policy within an agent loop must be carefully designed to account for the inherent uncertainty and potential noise in the evaluation process, ensuring that the agent only halts when it is genuinely confident that the desired quality threshold has been achieved.