{
  "id": 9467386,
  "title": "Why Detecting AI-Generated Text Is Harder Than You Think (And What I Built Anyway)",
  "url": "https://urgent.news/2026/09/24/why-detecting-ai-generated-text-is-harder-than-you-think-and-what-i",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-24T02:16:42.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/chenjiayan/why-detecting-ai-generated-text-is-harder-than-you-think-and-what-i-built-anyway-44lp"
  },
  "original_language": "en",
  "account": "The real challenge in detecting AI-generated text goes beyond what most AI detector websites claim. Promising a perfect 99% accuracy, these pages often mislead users by flagging their own work as generated by AI. This has led to students being unjustly accused, editors discarding human-written content, and a multitude of tools that appear to be guessing rather than providing accurate results. This prompted the author to delve into the intricacies of AI-content detection, construct their own tool to scrutinize these claims, and uncover the truth behind the marketing.\n\nWatermarking text presents a significant obstacle. Image models can embed an imperceptible statistical watermark, like SynthID, making it a verifiable signal. However, text behaves differently. Models generate tokens based on probability, leaving no natural space to embed a bit string that remains intact after copying and pasting. Researchers have attempted watermarking the logit distribution, but this method fails under paraphrasing, short inputs (a 20-word sentence carries little statistical footprint), translation, and deliberate obfuscation. Consequently, even tools that claim 100% accuracy on short text are deceiving users.\n\nWhen it comes to what AI models actually examine, their detectors rely on a few weaker statistical signals: perplexity, burstiness, and token-frequency patterns. Perplexity gauges how surprising the word choices are, and AI text tends to exhibit low perplexity, appearing smoother. Burstiness contrasts this by highlighting the variability in sentence length and rhythm between humans and LLMs. Lastly, n-gram and token-frequency patterns that reveal the decoding strategy. None of these serve as a reliable watermark. They are merely probabilities, and they falter when dealing with edited, mixed, or brief content. This discrepancy between statistical likelihood and verifiable provenance lies at the heart of most products' deceptive practices.\n\nDriven by the desire to bridge this gap, the author built a detector that delivers an honest confidence score coupled with an explanation of which signals triggered the assessment. Instead of offering a single false certainty, this tool provides transparent evidence. Interested users can explore the detector at https://detectaiwatermarks.com. In its technical implementation, the author emphasizes that relying solely on a naive check is insufficient; a useful approach yields meaningful evidence. The signals considered include perplexity (score_perplexity(text)), burstiness (sentence_variance(text)), and a scan for known watermarks (probe_known_watermark(text), which may return None, adding to the complexity of the analysis). The verdict is then calibrated according to content length, recognizing that a tweet differs significantly from an essay.",
  "summary": "The problem nobody talks about Every \"AI detector\" landing page promises 99% accuracy. Then you paste in a paragraph you actually wrote yourself and it flags you as ChatGPT. I kept seeing this in the wild — students wrongly accused, editors discarding human copy, and a pile of tools that were really just guessing. So I went down the rabbit hole of how AI-content detection actually works, built a…",
  "key_points": [
    "Detecting AI-generated text is harder than claimed by AI detector websites.",
    "Watermarking text is challenging due to token-based generation and lack of verifiable signal.",
    "Author built a detector providing honest confidence scores and explanation of triggering signals."
  ],
  "editors_take": "The author's development of a detector that provides transparent evidence and an honest confidence score marks a shift towards more reliable AI-generated text detection, diverging from the often misleading claims of existing tools.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}