{
  "id": 6456443,
  "title": "Better AI code comment detector",
  "url": "https://urgent.news/2026/09/09/better-ai-code-comment-detector",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-09T18:27:24.000Z",
  "source": {
    "name": "Lobsters",
    "slug": "lobsters",
    "url": "https://entropicthoughts.com/better-ai-comment-classifier"
  },
  "original_language": "en",
  "account": "A new AI comment classifier has been developed, utilizing public data for training and aiming to accurately detect whether code comments are written by humans or AI. The classifier achieved a balanced accuracy of 77%, meaning it correctly identifies human or computer-written comments 77% of the time. The tool also provides a predicted probability for each verdict, which is calibrated to represent the actual probability of the verdict being correct.\n\nConfusion matrix analysis reveals that the classifier makes mistakes in 25% of cases for both human and robot-generated comments, but the risk of a false positive is significantly lower when the classifier is confident (80% or more). Conversely, when the classifier is uncertain (around 50% confidence), the likelihood of an incorrect verdict is around 50%.\n\nThe classifier's performance has been validated through cross-validation, as well as manual testing on a smaller, real-world set of comments. The classifier appears to perform better with real-world data rather than synthetic data used in training. It's important to note that the classifier is specifically designed to detect AI-generated code comments, and its accuracy may vary for other types of text.\n\nThe development process involved collecting a dataset of code comments, primarily from permissively licensed or copyleft repositories. The dataset was then used to train the classifier, evaluating various features to optimize performance. However, the data collection and feature design process proved to be challenging and costly, with some errors requiring the regeneration of data.",
  "summary": "(I used both AI and vibecoding tags because the article describes training a logistic regression on LLM output. I thought there was a statistics tag that would have been appropriate but apparently not!) Comments",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}