Prompt search is a hill-climber, and accuracy is the wrong hill

The article discusses the limitations of optimizing prompts based on accuracy for tasks such as medical triage. The author shares an experience where a prompt scored highly on accuracy but performed poorly in real-world scenarios. The key point is that prompt optimization algorithms work by hill-climbing, meaning they find the easiest high point in the defined search space without understanding what that metric actually represents.

In the case of accuracy, it collapses scores into a binary yes/no, obscuring valuable ranking information. This leads to scenarios where different prompts can have the same accuracy but behave differently in practice, with the optimizer arbitrarily picking one or another. The article argues that accuracy is a poor hill to climb for imbalanced data, as it can lead to dangerous outputs.

A suggested solution is to modify the prompt optimization process to focus on metrics that better represent the deployment decision, such as AUROC (area under the receiver operating characteristic curve) instead of accuracy. This involves keeping raw scores instead of just boolean outcomes, computing AUROC alongside accuracy, and adjusting the search algorithm to match the intended decision-making process.

The author also notes potential downsides, such as increased computational cost and the challenge of handling imbalanced data at scale. Overall, the key takeaway is that the metric used by prompt optimization should align with the actual metric used in deployment, as prompting algorithms are honest about finding the best solution according to the specified criteria.