meta-externalagent and meta-webindexer: crawlers were a third of my visitors
meta-externalagent and meta-webindexer are Meta's web crawlers, and they run JavaScript. That means your client-side analytics records them as visitors. On my site they looked like a desktop Chrome at 1920x1080, browsing from US datacenters, and they were 40 of the 43 visitors who hit an error in one week. With AhrefsBot and ShapBot, crawlers made up 31% of all "visitors". They never converted. I…
Meta's crawlers, meta-externalagent and meta-webindexer, accounted for one-third of the visitors to a reporter's website. These crawlers run JavaScript, making them appear as desktop Chrome visitors in client-side analytics. In one week, 40 out of 43 visitors who encountered an error were these crawlers, which made up 31% of all visitors along with AhrefsBot and ShapBot.
These crawlers were not filtered out by the site's existing bot filters, leading to these false conversions. The report found that these crawlers often visit from US datacenters, unlike the reporter's actual readers, and do not convert on the site. To address this issue, the reporter added a regular expression to filter these crawlers out before sending any beacon.
After implementing the filter, the number of visitors dropped from 409 to 265, and the number of exceptions from 48 to 6. The remaining exceptions were from a real visitor using Firefox, whose blocked chat widget was now counted as an event rather than an exception. The reporter suggests that while blocking these crawlers might not be necessary, filtering them out of analytics is crucial to accurately count visitors and maintain open access to the site.
Rewritten by urgent.news; may contain errors — check the original.
Source: dev.to