Why Detecting AI-Generated Text Is Harder Than You Think (And What I Built Anyway)

The real challenge in detecting AI-generated text goes beyond what most AI detector websites claim. Promising a perfect 99% accuracy, these pages often mislead users by flagging their own work as generated by AI. This has led to students being unjustly accused, editors discarding human-written content, and a multitude of tools that appear to be guessing rather than providing accurate results.

This prompted the author to delve into the intricacies of AI-content detection, construct their own tool to scrutinize these claims, and uncover the truth behind the marketing.

Watermarking text presents a significant obstacle. Image models can embed an imperceptible statistical watermark, like SynthID, making it a verifiable signal. However, text behaves differently. Models generate tokens based on probability, leaving no natural space to embed a bit string that remains intact after copying and pasting.

Researchers have attempted watermarking the logit distribution, but this method fails under paraphrasing, short inputs (a 20-word sentence carries little statistical footprint), translation, and deliberate obfuscation. Consequently, even tools that claim 100% accuracy on short text are deceiving users.

When it comes to what AI models actually examine, their detectors rely on a few weaker statistical signals: perplexity, burstiness, and token-frequency patterns. Perplexity gauges how surprising the word choices are, and AI text tends to exhibit low perplexity, appearing smoother. Burstiness contrasts this by highlighting the variability in sentence length and rhythm between humans and LLMs.

Lastly, n-gram and token-frequency patterns that reveal the decoding strategy. None of these serve as a reliable watermark. They are merely probabilities, and they falter when dealing with edited, mixed, or brief content. This discrepancy between statistical likelihood and verifiable provenance lies at the heart of most products' deceptive practices.

Driven by the desire to bridge this gap, the author built a detector that delivers an honest confidence score coupled with an explanation of which signals triggered the assessment. Instead of offering a single false certainty, this tool provides transparent evidence. Interested users can explore the detector at https://detectaiwatermarks.com.

In its technical implementation, the author emphasizes that relying solely on a naive check is insufficient; a useful approach yields meaningful evidence. The signals considered include perplexity (score_perplexity(text)), burstiness (sentence_variance(text)), and a scan for known watermarks (probe_known_watermark(text), which may return None, adding to the complexity of the analysis).

The verdict is then calibrated according to content length, recognizing that a tweet differs significantly from an essay.