AI models catch bad code, then cry wolf on the good code

I conducted a Kaggle benchmark called Blog vs Bytecode to explore how AI models handle data-science tutorials. Each pair featured a piece of code alongside a corresponding blog-style claim. Half of the claims were genuine, while the other half contained deliberate mistakes like improper scaler fitting, feature construction from the target variable, improper cross-validation on time-series data, quoting accuracy on highly imbalanced data, and threshold tuning on the test set.

The dataset was balanced at 15/13, meaning a model that simply flagged every example as problematic would achieve a 50% accuracy score. I tested various current models on Kaggle and observed that some models were excellent at identifying genuine flaws, but they also often flagged correct code as problematic. Conversely, other models tended to miss the flawed code.

I discovered that the issue stemmed from the harness used to run the models on Kaggle. The harness was lying about half the models it ran for you, leaving empty responses which were counted as wrong. When I fixed the harness by having it return the actual model output instead, the performance of the models improved significantly. For instance, DeepSeek-R1's error rate dropped from 17% to 100% when the harness was fixed.

While the strong models were near-perfect at flagging genuine flaws, they also tended to call sound code problematic. Small models, on the other hand, under-flagged flaws. Notably, models with reasoning capabilities scored better when reasoning was enabled, while those without reasoning saw a significant drop in performance. The benchmark, which includes all items, verdicts, and the grader, is publicly available on Kaggle for anyone to examine the models' outputs and identify any errors.