Better AI code comment detector
(I used both AI and vibecoding tags because the article describes training a logistic regression on LLM output. I thought there was a statistics tag that would have been appropriate but apparently not!) Comments
A new AI comment classifier has been developed, utilizing public data for training and aiming to accurately detect whether code comments are written by humans or AI. The classifier achieved a balanced accuracy of 77%, meaning it correctly identifies human or computer-written comments 77% of the time. The tool also provides a predicted probability for each verdict, which is calibrated to represent the actual probability of the verdict being correct.
Confusion matrix analysis reveals that the classifier makes mistakes in 25% of cases for both human and robot-generated comments, but the risk of a false positive is significantly lower when the classifier is confident (80% or more). Conversely, when the classifier is uncertain (around 50% confidence), the likelihood of an incorrect verdict is around 50%.
The classifier's performance has been validated through cross-validation, as well as manual testing on a smaller, real-world set of comments. The classifier appears to perform better with real-world data rather than synthetic data used in training. It's important to note that the classifier is specifically designed to detect AI-generated code comments, and its accuracy may vary for other types of text.
The development process involved collecting a dataset of code comments, primarily from permissively licensed or copyleft repositories. The dataset was then used to train the classifier, evaluating various features to optimize performance. However, the data collection and feature design process proved to be challenging and costly, with some errors requiring the regeneration of data.
Written by urgent.news from Lobsters's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
