I crawled 3,014 Houston business websites to see what AI crawlers actually see
Everyone keeps asking me if they should block ChatGPT from their website. So I went and looked at what businesses in Houston are actually doing, and the answer surprised me: almost nobody is blocking AI. Their sites just aren't built in a way AI can read. Here's what I did and what I found. The full dataset is public if you want to poke at it [links at the bottom]. The setup I pulled every…
A journalist analyzed 3,014 Houston business websites to determine what AI crawlers actually see. The study found that almost no businesses had blocked AI crawlers, despite many believing it was necessary. The website layout made it difficult for AI to read the content. Only 1.6% of websites blocked AI crawlers in their robots.txt file, but when tested using a normal browser user agent, the number was closer to 5%.
The key to making a website AI-friendly is to ensure that the main content is rendered on the server and that there are at least 120 words of readable text before any JavaScript runs. Additionally, having a LocalBusiness schema in JSON-LD and an H1 heading also help.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
