curl your own homepage. That is all ChatGPT sees.
Run this against your site right now: curl -s https://yoursite.com | grep -o "<h1[^>]*>.*</h1>" If nothing comes back, or you get an empty <div id="root"> , then large parts of the internet cannot read your site. Not "reads it poorly". Cannot read it. I do this on every site we take over, and the result surprises people often enough that it is worth writing down. What the test is actually showing…
You can quickly assess if search engines can read your website using the command `curl -s https://yoursite.com | grep -o h1[^ ]* .* /h1`. If the output is empty or you receive `div id=root`, it indicates that major parts of the internet cannot read your site. This is because the test fetches HTML and stops, without executing JavaScript or waiting for hydration. Large portions of automated readers, such as GPTBot, ClaudeBot, and PerplexityBot, do not run JavaScript and primarily focus on fetching the HTML.
To ensure your content is visible to automated crawlers, consider the following steps:
1. Analyze the raw HTML by counting the words using `curl -s https://yoursite.com | wc -c` and `sed` commands to remove scripts and whitespace. A marketing homepage with only a few words of real text in the raw HTML is a red flag.
2. Compare the raw HTML word count to the rendered content by examining the length of the text inside the body element in the browser's Developer Tools. A significant difference between the two indicates that content only appears after JavaScript runs.
3. Test with the appropriate user agent, such as `GPTBot`, to verify if the crawler can access your key phrases and content.
4. Ensure your robots.txt file does not accidentally block crawlers by checking `curl -s https://yoursite.com/robots.txt`.
Fixing the issue typically involves ensuring the important content is present in the first HTML response. This may require reviewing your framework setup, such as Next.js or Vite, and making sure server components include the necessary HTML and text. By following these steps, you can guarantee that your website's content is accessible to both human readers and automated crawlers.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
