9 Bugs That All Looked Like a Working System

AgentSelfEdit is an open-source tool that automatically iterates on its own system prompt based on execution feedback. It performs A/B testing on edits and only adopts the statistically-proven winners. However, while trying it out, I discovered nine critical bugs that all produced output that looked correct but was actually flawed.

Bug 1: The Gate Letting Noise Through as Improvement

The "promotion gate" was incorrectly checking for statistical significance. Instead of looking for a p-value below 0.05, it was checking for a p-value above 0.95. This meant that edits with even a 90% chance of being random noise were being accepted as improvements. After fixing this, the same edit that previously passed with p=0.9 now only passed with p=0.23, rejecting 23% of improvements as noise.

Bug 2: A/B Test Compared Prompt Against Itself

The A/B testing logic compared the edited prompt against the original prompt, rather than the edited version against the unedited version. When the A/B test produced a perfect tie (p=1.0), it incorrectly rejected the improvement. This occurred because the system was editing only a fragment of the prompt rather than the full system prompt. After fixing the code to construct the full candidate prompt from the edited fragment, the A/B test generated a correct p-value and rejected the improvement when appropriate.

Bug 3: Scoring Mode Marked Everything as Passed

There was a "label" scoring mode designed for real trace data, which incorrectly marked any trace with any non-empty response as a pass. This meant that even failure traces were scored as 100% correct. To fix this, the label scoring mode was entirely removed from the system.

Bug 4: Docker Tests Skipped the Promotion Gate

The Docker integration tests were configured to bypass the promotion gate and A/B testing. While running a dry-run, the system failed to promote the actual improvement because these tests didn't go through the proper testing gates. This created a false sense of security that the system was working correctly.

Bug 5: Failure Traces Were Fabricated

The failure traces were not being generated correctly and were being fabricated, leading to incorrect conclusions about the system's performance. This bug was challenging to identify as it had subtle effects on the system metrics.

Bug 6: The Gate Received the Wrong Prompt

The promotion gate was receiving an incorrect system prompt instead of the correct one, causing it to evaluate the wrong prompt and produce misleading results.

Bug 7: The CLI Talked to a Mock Instead of a Real LLM

The command-line interface (CLI) was configured to interact with a mock LLM rather than the actual system, giving a false impression of the system's performance.

Bug 8: The Config Silently Ignored the Endpoint

The configuration file had a bug that caused it to ignore the endpoint specified for the LLM, leading to the system using a default or improperly configured endpoint.

Bug 9: The Field Test Runner Measured the Wrong Thing

The field test runner was measuring incorrect metrics, focusing on irrelevant aspects of the system's performance and neglecting the key metrics that truly indicated its effectiveness.

Each of these bugs, while seemingly minor, had significant consequences on the system's accuracy and reliability. They highlighted the importance of thoroughly testing and validating every component of an AI system, especially when it is designed to self-improve. The experience taught me the critical importance of rigorous testing, proper statistics, and careful implementation when building systems on top of large language models.