My Comment Section Designed My Next Experiment. Then It Made Me Freeze My Predictions.

Ten days after publishing an article about a language model's failure mode, I received feedback from readers that transformed the study. The comment section scrutinized my experiment, leading to a more rigorous approach. I first ran the study with the original setup, but the feedback prompted me to write down and publish my predictions before conducting the experiment.

This change ensured the results would be transparent and credible. The experiment used the same dataset, model configurations, and methods, with key differences being the inclusion of a neutral control arm and a preregistration of predictions. The results showed significant differences in confirmation rates between the models, particularly with GPT-4o-mini confirming 90% of the code as vulnerable, even without a flag.

This led to further statistical analysis, which revealed a significant difference in the flipping of confirmations and rejections between the models, indicating that GPT-4o-mini was more prone to over-reporting vulnerabilities. The experiment's results were published alongside the predictions, adhering to the pre-defined decision rules and thresholds.

This approach allowed for a transparent and unbiased evaluation of the language models' performance in detecting vulnerabilities.