AI slop detector works well on text but much less so on images: Here’s why

In an era of rampant AI-generated content, a new AI tool, Pangram, has emerged to detect whether words have been churned out by chatbots or written by humans. By analyzing the stylistic fingerprint of AI models, Pangram boasts near-perfect accuracy in identifying AI-generated text. However, detecting AI-generated images remains challenging for the tool.

Testing the efficacy of Pangram, a reporter conducted numerous experiments using various passages of personal writing, passages generated by AI, and publicly available AI-generated content. In most cases, Pangram accurately distinguished human writing from AI-generated text. For instance, it correctly identified sentences generated by a chatbot within a personal essay and pinpointed AI-generated components in LinkedIn posts. Even passages from Charles Dickens were deemed 100% human-written by the tool.

Despite its impressive performance in detecting AI-generated text, Pangram has limitations when it comes to identifying AI-generated images. Tests using 20 widely shared AI-generated images revealed that the tool failed to correctly identify a majority of the images as AI-generated, including a deepfake of actress Zendaya appearing pregnant, a photo of Sen. Mitch McConnell on a hospital bed, and a photo depicting a San Francisco street sign suggesting it was okay to steal goods under $950.

While Pangram's image detection capabilities are currently less accurate than its text analysis, the tool still flagged some infamous AI-generated fakes, such as a photo of Zendaya and Tom Holland's wedding, a picture of President Trump with a child during his China visit, and a photo of the Clintons partying with Jeffrey Epstein. However, the tool declined to analyze certain violent or low-quality images, highlighting its limitations in image-based content detection.

As AI-generated content proliferates on the web, AI detection tools like Pangram may become essential in protecting users from deceptive and low-quality information. Nevertheless, the technology for identifying AI-generated images still lags behind its text-based counterpart, raising concerns about the potential for misinformation to spread through visual media.