Professors Are Tearing Their Hair Out Over AI Detectors

Timothy Paustian, a biology professor at the University of Wisconsin at Madison, has long struggled with students using AI to write their essays. When ChatGPT first appeared, Paustian tried various AI detectors, but they were unreliable. He then added covert prompts that only AI could respond to, which led him to catch around 60 AI-written essays out of 350 students in his online microbiology class last year.

However, the recent emergence of a tool called Pangram, which has a very low false-positive rate, has left Paustian back where he started. The hidden prompts that Paustian used to catch AI have been removed by students, and his university discourages professors from relying solely on AI detectors. This year, Paustian plans to eliminate writing assignments altogether, despite writing being crucial for learning.

AI detectors are becoming increasingly sophisticated, but they are not foolproof. The MIT working group recently recommended against relying on AI detectors due to the risk of creating a mistrust between instructors and students. Similarly, Turnitin, the most widely used AI detection tool in higher education, has come under scrutiny for its high false-positive and false-negative rates.

Turnitin claims its false positive rate is less than 1%, but this translates to potentially 750 students being mistakenly flagged out of 75,000 student papers annually. Pangram, on the other hand, boasts a false-positive rate of one in 25,000, but it has yet to gain traction in academic circles. Universities like Wisconsin and Vanderbilt have relied on outdated research that concluded early AI detection tools were ineffective and have not updated their policies.

Despite the concerns, professors and institutions are still grappling with how to address AI cheating in an increasingly challenging landscape.