Urgent.News

the world's headlines, one feed

Tech

A page my crawler could not read is not a page that does not exist. Six times I got that wrong, twice while writing this

I have been running a small measurement rig for two weeks: a set of scripts that load our pages in a fresh anonymous browser context and record what is actually there. Directory listings, articles, search positions. It has been genuinely useful and it has been wrong in one specific way, six separate times, and the six are the same mistake. Every time, a script failed to read something, and I…

We haven't written up this one. Dev.to has the full story — the link below goes straight to it.

Read the original at dev.to →

More in Tech