I built a local heuristic to catch AI sounding product copy. Here is what it actually catches and what it misses.
This week I packaged a small analyzer I had been testing as a Chrome extension: Copy Tell. It checks product descriptions and posts on Gumroad and Substack for the kind of stock phrasing an unedited AI writing tool tends to produce, and shows a small notice if enough of it shows up. No ML model. Pure lexical and stylometric scoring: stock phrase density, symmetric list framing used as filler,…
A Chrome extension called Copy Tell has been developed to identify AI-generated product descriptions on marketplaces like Gumroad and Substack. The tool works by analyzing the text for patterns commonly found in machine-generated content, such as stock phrases, symmetric list framing, and sentence length uniformity. It runs entirely in the browser, without sending any data outside the webpage.
To evaluate its performance, Copy Tell was tested against a dataset of 100 product descriptions - 50 AI-generated and 50 human-written. The results showed 39 true positives, 1 false positive, 49 true negatives, and 11 false negatives. The precision of the tool was strong at 97.5%, meaning that when it flagged a description as potentially AI-generated, it was correct 97.5% of the time. However, the recall was lower at 78%, indicating that the tool missed 22% of the AI-generated samples.
The 11 false negatives shared a common characteristic - they were written in a more straightforward, literal style, lacking the typical stock phrases that the analyzer is designed to catch. This limitation stems from the fact that a rule-based approach like Copy Tell cannot easily incorporate natural language understanding, which would be necessary to improve recall.
One false positive was identified: the README for the axe-core library was flagged primarily due to the presence of a single phrase that a human author also used. While this is a fair miss, it's not a case of the tool being cherry-picked for an easy target.
It's important to note that the test set primarily consisted of AI-generated copy versus real technical documentation, rather than actual marketplace listings. In real-world scenarios, where sellers often use more superlative language, the tool's precision might be lower than the 97.5% reported in the test.
The extension is currently live and free to use at https://chromewebstore.google.com/detail/copy-tell-ai-description/ehpkbnkcpbfkagcfmbofldfnpnmopcbi. The developer encourages users to try it out on their own listings and report any errors, whether false positives or false negatives, to help improve the tool's accuracy.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.