Urgent.News

What's breaking now, across thousands of outlets.

More in Tech

Shipping a 592-URL site in 8 locales as a static export: what actually broke

I build for the web. The thing I most recently shipped is Solitaire Online : 27 solitaire variants, eight languages, no third-party ad scripts, exported as static HTML.

  • The website Solitaire Online has 592 URLs across eight languages.
  • The site uses a single shared slug for each game, not separate slugs per language.
  • Challenges included sitemap weight, icon assets, and screenshot routes.

A robots.txt that allows GPTBot does not mean GPTBot can reach your site

I built SeenLens, a one-time AI readiness audit for websites. I wanted to know what happens when a page's published crawler rules and its HTTP responses tell different stories. I scanned one public URL on each of 30 sites. In that run, 5 returned HTTP 403 or 202 to the scanner.

  • Robots.txt allowing GPTBot doesn't guarantee access.
  • Server responses can conflict with robots.txt directives.
  • CDN or WAF may block GPTBot access despite allowed tokens.

More from Saturday 7 November →