41 Experiments in 6 Days: What the Human Was Actually For

In a six-day experiment, an AI agent named Claude conducted 41 experiments to determine the purpose of a human's involvement. The initial F1 score was 0.638, a significant improvement over the previous best of 0.598. The agent's primary role was to generate hypotheses and implement them rapidly, as the evaluation loop allowed for quick decision-making on whether to keep or revert changes. However, it was the humans who identified when the metrics were misleading.

In the early stages, the agent focused on removing friction by implementing non-deferred tools, date formatting in search results, and structured output. These changes improved the F1 score from 0.392 to 0.523. As the experiments progressed, the agent moved on to fact extraction, speaker trees, and previous/next linking, which slightly increased the F1 score from 0.523 to 0.562.

The most significant improvements came after recalibrating the evaluation framework and removing the category hint from the prompt. This change, although seemingly minor, significantly impacted the results. The agent's performance dropped from 0.659 to 0.642, indicating that the human played a crucial role in ensuring that the AI was measuring the right thing.

Throughout the process, the agent excelled at tasks that scaled with volume, such as reading failing questions and identifying bugs invisible to human evaluation. However, the human was better suited for determining whether the metrics were accurate and identifying the simplest explanation for the results. This division of labor proved faster than either party working independently, as the human caught errors that the agent could not identify.

Ultimately, the experiments showed that while the agent is effective at automating certain tasks, it is the human who plays a vital role in ensuring the AI is measuring the intended objectives accurately. The six-day experiment demonstrated the value of combining AI and human intelligence to achieve better results.