Spotting AI writing: how reliable are the detectors?

Canadian-Haitian writer Thelyson Orelien is facing allegations that his best-selling debut novel was partially written using AI tools. The claim originated from an X account that touted the results of the AI-detection program Pangram, which it asserted had determined that the story of a Haitian migrant traveling to Canada had been largely penned by a text-generator like Claude or ChatGPT.

Orelien, 38, vigorously denies the allegations and vows to provide proof that he authored the book, which has been translated into more than 20 languages and sold to publishers worldwide.

Multiple AI detection tools were employed to scrutinize excerpts from Orelien's novel, yielding mixed results ranging from a high likelihood of human authorship to a definitive AI-generated conclusion. These digital detectors, which are increasingly prevalent in educational and publishing settings, typically compare AI-generated text against a vast database of human-written content to identify patterns and linguistic cues that distinguish one from the other.

Research published in October 2025 by the University of Chicago, titled "Artificial Writing and Automated Detection," revealed that Pangram, the program at the center of the controversy, has remarkably low error rates when examining medium-to-long text samples. Thierry Poibeau, a researcher at the CNRS and an expert in natural language processing, explained that these detection systems are designed to identify "linguistic tics" such as triadic phrasing (the use of three clauses) or em dashes, which were once prevalent in AI-written texts due to their training data being rooted in American scientific writing or novels.

Despite the progress made by detection tools, their accuracy remains a contentious issue. The percentages provided by these programs are difficult to validate and do not account for the potential for false positives. Max Spero, co-founder of Pangram, cautioned against relying solely on such tools, suggesting they should act as a preliminary indicator rather than the final verdict in cases of suspected AI-generated content.

Poibeau also highlighted the ongoing challenge posed by the constantly evolving nature of AI programs, which could render detection methods obsolete quite swiftly.

Orelien's defense hinges on his distinctive writing style, which he attributes to Haitian and Caribbean traditions. He told Liberation newspaper, "I didn't write this novel with my head, but with my guts and my heart." According to Thierry Poibeau, the effectiveness of these detection tools may be compromised when applied to novels, as they might detect repetitive patterns that are actually indicative of an author's unique voice or style.

For instance, Orelien claimed that the software employed was attuned to the French literary tradition and classical authors.

Benoit Raphael, an entrepreneur and AI enthusiast, conducted his own investigation using Pangram to test Orelien's work and found results that were heavily in favor of AI authorship, ranging from 90 to 100 percent. This high AI detection rate is seen by Raphael as a "pretty telling sign." While the novel received widespread acclaim from critics and sold briskly in France prior to the controversy, several critics began suspecting AI involvement due to repetitive sentence structures and an overabundance of analogies, which are often associated with AI-generated prose.

Notably, Lea Bory, a podcaster, was among the first to publicly suggest, on her show Torchon, that the novel's sentence repetition and excessive use of analogies, hallmarks of AI-generated text, led her to believe it was written by an artificial intelligence. In Canard Enchaine, Fabrice Colin also criticized the formulaic sentence structure and lamented the lack of a distinctive voice in the prose.