{
  "id": 11817556,
  "title": "I crawled 3,014 Houston business websites to see what AI crawlers actually see",
  "url": "https://urgent.news/2026/10/04/i-crawled-3-014-houston-business-websites-to-see-what-ai-crawlers",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-04T01:47:54.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/marketingandai/i-crawled-3014-houston-business-websites-to-see-what-ai-crawlers-actually-see-1fj7"
  },
  "original_language": "en",
  "account": "A journalist analyzed 3,014 Houston business websites to determine what AI crawlers actually see. The study found that almost no businesses had blocked AI crawlers, despite many believing it was necessary. The website layout made it difficult for AI to read the content. Only 1.6% of websites blocked AI crawlers in their robots.txt file, but when tested using a normal browser user agent, the number was closer to 5%. The key to making a website AI-friendly is to ensure that the main content is rendered on the server and that there are at least 120 words of readable text before any JavaScript runs. Additionally, having a LocalBusiness schema in JSON-LD and an H1 heading also help.",
  "summary": "Everyone keeps asking me if they should block ChatGPT from their website. So I went and looked at what businesses in Houston are actually doing, and the answer surprised me: almost nobody is blocking AI. Their sites just aren't built in a way AI can read. Here's what I did and what I found. The full dataset is public if you want to poke at it [links at the bottom]. The setup I pulled every…",
  "key_points": [
    "Journalist analyzed 3,014 Houston business websites",
    "Only 1.6% blocked AI crawlers in robots.txt",
    "Key to AI-friendliness: server-rendered content and readable text"
  ],
  "editors_take": "Most Houston businesses inadvertently hinder AI crawlers with complex website layouts, despite few actively blocking them, suggesting a need for AI-friendly design adjustments.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}