Urgent.News

What's breaking now, across hundreds of outlets.

Tech

meta-externalagent and meta-webindexer: crawlers were a third of my visitors

meta-externalagent and meta-webindexer are Meta's web crawlers, and they run JavaScript. That means your client-side analytics records them as visitors. On my site they looked like a desktop Chrome at 1920x1080, browsing from US datacenters, and they were 40 of the 43 visitors who hit an error in one week. With AhrefsBot and ShapBot, crawlers made up 31% of all "visitors". They never converted. I…

Meta's crawlers, meta-externalagent and meta-webindexer, accounted for one-third of the visitors to a reporter's website. These crawlers run JavaScript, making them appear as desktop Chrome visitors in client-side analytics. In one week, 40 out of 43 visitors who encountered an error were these crawlers, which made up 31% of all visitors along with AhrefsBot and ShapBot.

These crawlers were not filtered out by the site's existing bot filters, leading to these false conversions. The report found that these crawlers often visit from US datacenters, unlike the reporter's actual readers, and do not convert on the site. To address this issue, the reporter added a regular expression to filter these crawlers out before sending any beacon.

After implementing the filter, the number of visitors dropped from 409 to 265, and the number of exceptions from 48 to 6. The remaining exceptions were from a real visitor using Firefox, whose blocked chat widget was now counted as an event rather than an exception. The reporter suggests that while blocking these crawlers might not be necessary, filtering them out of analytics is crucial to accurately count visitors and maintain open access to the site.

Rewritten by urgent.news; may contain errors — check the original.

Source: dev.to

Read the original at dev.to →

Report a problem with this story

More in Tech

I built KaggleFleet to simplify collaborative Kaggle GPU workflows

Working on ML experiments with multiple collaborators often means dealing with multiple notebook runs, Kaggle accounts, and output files.

  • KaggleFleet simplifies collaborative Kaggle GPU workflows
  • Local-first notebook workflow retains notebooks in project
  • Multiple accounts configured independently via browser auth

Why Your Headless Browser Can't Scrape Everything

Originally published on my blog — read it there with the full context, code and updates . I hit the same wall a few weeks apart, on two different second-hand marketplaces, and the contrast taught me…

  • Headless browsers scrape Carousell's freshly-listed products easily.
  • Goofish blocks headless browsers, requiring authentication and anti-bot checks.
  • Data source determines scraping approach: server-rendered vs asynchronous requests.

More from Sunday 22 November →