Build Buddy, Day 2: two AI fixes compiled, only one was correct

Day 2 of the MLH Global Hack Week Builders Week saw my AI-powered tool Build Buddy successfully fix six out of six simple C# errors using a local model. To compare models on a more challenging task, I set up a C# file with three intentional errors, including CS0029, CS1061, and CS0161.

Before running any models, I followed a strict protocol: I documented the exact prompt, input file, scoring rules, and the compiler's output for the broken file. I then built each answer in a clean copy of the project to ensure a fair comparison.

My first attempt used a missing semicolon instead of the typo, which the baseline compiler caught, stopping the compilation process and not reporting the other two errors. To avoid this issue, I replaced the typo with the correct one before running any models.

The results were mixed. Grok 4.7 (Cursor) successfully compiled the code, fixed all three errors, and named all three error codes correctly. It took 33 seconds to complete. On the other hand, Gemma 3 4B, a local model, was twice as fast but made several mistakes. It used .Result instead of await, which blocked the thread, rewrote the Half() function to return incorrect results, added an unnecessary using System; directive, and invented an error about System.Linq being unused. The explanation provided by Gemma also failed to mention the Task / await problem.

The takeaway from this experiment is that compiling code is necessary but not sufficient. The behavior and explanations provided by the models should also be carefully evaluated. It's crucial to record a baseline before testing any fixes and start every check from a clean folder to ensure accurate results.