I pre-registered a study on AI visibility signals. The main result was null.
Originally published on angeo.dev . Full tables, p-values and the sealed plan are there. Most claims about AI visibility are untestable by design: publish the signals, wait, attribute anything good that happens to the signals. I wanted a version I could not fudge, so I wrote the analysis plan first, hashed it, and sent the hash to the other party before I had any data. The question Do businesses…
A researcher has pre-registered a study examining whether businesses that repeatedly mention AI assistants differ from those that mention them only once, based on observable technical signals. Four signals were considered: crawler access, serving llms.txt, having a product page with JSON-LD structured data, and offering product availability data.
The study used data from 458 product-level home-decor purchasing questions across three AI assistants (ChatGPT, Gemini, and Perplexity). Blinding was employed, ensuring the researcher did not see the store list or the analysis frame until the plan was sealed. The analysis found no signal to separate the two groups by more than 15 points.
Specifically, llms.txt was more common among businesses mentioned once rather than repeatedly, and this difference was found to be consistent across all five cuts of the data. However, the apparent difference was attributed more to boilerplate content rather than authentic adoption. The researcher stresses the importance of pre-registering failure conditions, separate inspection of content, and considering alternative sampling heuristics to avoid manufacturing schema changes.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
