{
  "id": 4220574,
  "title": "curl your own homepage. That is all ChatGPT sees.",
  "url": "https://urgent.news/2026/08/29/curl-your-own-homepage-that-is-all-chatgpt-sees",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-29T15:36:27.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/techchoom/curl-your-own-homepage-that-is-all-chatgpt-sees-3kpm"
  },
  "original_language": "en",
  "account": "You can quickly assess if search engines can read your website using the command `curl -s https://yoursite.com | grep -o h1[^ ]* .* /h1`. If the output is empty or you receive `div id=root`, it indicates that major parts of the internet cannot read your site. This is because the test fetches HTML and stops, without executing JavaScript or waiting for hydration. Large portions of automated readers, such as GPTBot, ClaudeBot, and PerplexityBot, do not run JavaScript and primarily focus on fetching the HTML.\n\nTo ensure your content is visible to automated crawlers, consider the following steps:\n1. Analyze the raw HTML by counting the words using `curl -s https://yoursite.com | wc -c` and `sed` commands to remove scripts and whitespace. A marketing homepage with only a few words of real text in the raw HTML is a red flag.\n2. Compare the raw HTML word count to the rendered content by examining the length of the text inside the body element in the browser's Developer Tools. A significant difference between the two indicates that content only appears after JavaScript runs.\n3. Test with the appropriate user agent, such as `GPTBot`, to verify if the crawler can access your key phrases and content.\n4. Ensure your robots.txt file does not accidentally block crawlers by checking `curl -s https://yoursite.com/robots.txt`.\n\nFixing the issue typically involves ensuring the important content is present in the first HTML response. This may require reviewing your framework setup, such as Next.js or Vite, and making sure server components include the necessary HTML and text. By following these steps, you can guarantee that your website's content is accessible to both human readers and automated crawlers.",
  "summary": "Run this against your site right now: curl -s https://yoursite.com | grep -o \"<h1[^>]*>.*</h1>\" If nothing comes back, or you get an empty <div id=\"root\"> , then large parts of the internet cannot read your site. Not \"reads it poorly\". Cannot read it. I do this on every site we take over, and the result surprises people often enough that it is worth writing down. What the test is actually showing…",
  "key_points": [
    "curl command reveals if search engines can read website",
    "Major parts of internet may not read site without JavaScript",
    "Fix by ensuring important content in first HTML response"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}