A reader found the flaw in my code-graph test
I published a result that said my tool scored 34.5% and another scored 5%. A reader on Reddit asked two questions about how I counted. I went back, fixed the method, tested 40 more commits, and the result didn't hold. This is what went wrong, and how I'd run this kind of test now. The test I build EngramGraph, a code graph. I wanted to know if a code graph can tell an AI agent which other files…
A reader identified an error in the author's code-graph testing. Initially, the author claimed their tool, EngramGraph, outperformed another tool, code-review-graph, by a significant margin. However, after receiving feedback, the author discovered the issue and retested using a different method. The main problems were that empty predictions were being treated as incorrect and the author hadn't considered the potential for other files to change together without function calls linking them.
After fixing these issues, the author found that EngramGraph and code-review-graph performed more similarly, with EngramGraph having a 35% accuracy and code-review-graph having a 51% accuracy when retries were used. The author also realized that the potential for a tool to be correct was lower than initially assumed and recommended reporting the frequency of tool answers separately from their accuracy.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
