Spotting AI writing: How reliable are the detectors?

The French literary world is buzzing following claims that Canadian-Haitian author Thelyson Orelien allegedly used AI tools to write his bestselling debut novel. The controversy erupted after an X account suggested that Pangram, an AI detection program, concluded that the story of a Haitian migrant's journey to Canada was mostly written by a generative text tool. Orelien, 38, firmly denies the allegations and vows to prove he penned the work, which has been published in over 20 countries.

Various AI detection tools were employed to examine excerpts from the book, yielding mixed results. Some tools suggested the text was "very likely" or even "100%" AI-generated, while others were more skeptical. These tools typically function by comparing the input text to large databases of human-written and AI-generated content, scanning for discrepancies.

The tools have shown remarkable improvement in recent years, with one study from the University of Chicago indicating Pangram can achieve near-zero error rates on medium-to-long text samples.

Authors and researchers have identified several clues that may indicate AI involvement. These include triadic phrasing (using three clauses) and the overuse of dashes, which were more common in generated texts due to the training data's origin. Pangram claims its system can detect AI text by "analysing and understanding subtle cues in the writing." Some writers exhibit telltale patterns, such as repetitive sentence structures and monotonous phrasing.

However, the reliability of these tools remains contentious. Many publish percentage scores for accuracy, but these figures cannot be independently verified and don't account for false positives. Pangram's CEO and co-founder, Max Spero, has warned that the tool should not be the final word but rather a starting point for further investigation.

Thierry Poibeau, a researcher at the CNRS, noted that AI programs "constantly evolve," making these detection systems a "moving target problem." He also pointed out that AI-generated text may possess a writer's recognizable style, which could be misinterpreted by detection systems.

Despite the controversy, Orelien insists that his writing style is rooted in Haitian and Caribbean traditions. He told AFP that the AI was "used to the French literary tradition and classical authors," suggesting that the tool may have inadvertently replicated his unique voice. Other experts, such as Benoit Raphael, a specialist in AI, tested Pangram's findings on Orelien's work and also found evidence of AI involvement. Raphael believes the high AI detection scores are indicative of the program's accuracy.

Public skepticism about the novel's authenticity has been simmering since its release, with critics and readers alike expressing concerns about the overuse of analogies and repetitive sentence structures. Lea Bory, on her podcast "Torchon," was among the first to voice her suspicions, citing the formulaic nature of Orelien's prose. In the magazine Canard Enchaine, Fabrice Colin criticized the novel for its lack of a distinctive voice, urging authors to embrace a more authentic writing style.