Meta’s web crawlers are making life hell for owners of small business websites

Meta Platforms, the parent company of Facebook and Instagram, has come under criticism for its upcoming changes to its terms that allow AI crawlers to freely access websites. However, Meta has been employing AI crawlers on small business websites for years, leaving these businesses to deal with the consequences. This was recently demonstrated when Meta's crawlers crashed an artist's website by generating over 550,000 hits in just a few days.

The crawlers, such as Meta-ExternalAgent and Meta-WebIndexer, are designed to train AI models and improve search result quality. However, website owners claim that Meta's lack of guardrails leaves them vulnerable to these bots. For instance, Paulus initially tried to block the crawlers by setting up a firewall and updating her robots.txt file, but neither method was successful. It was only when her host provided specific code for the website's .htaccess file that the bots were blocked.

Robyn Gough, a small digital design business owner in Australia, faced similar issues. In October, her servers logged 2.45 TB of data delivered, with 97.6% of it coming from Meta's crawlers - mostly Meta-ExternalAgent and later Meta-WebIndexer. This excessive traffic led to server errors and disrupted her ability to deliver products to customers.

Both Paulus and Gough have chosen not to share their websites publicly to avoid further unwanted traffic. Gough discovered that the Meta-ExternalAgent crawler had been accessing her website since August 2024, but it wasn't until November 2025 that the traffic significantly increased. She had to pay a $10.51 fee to her host after exceeding the 2 TB limit for her virtual private server.

These small business owners emphasize the challenges they face in dealing with these automated crawlers. For many, it's an issue of limited resources and knowledge - they lack developers and security teams to manage these incidents. As a result, they may be forced to take their websites offline or pay additional fees to accommodate the traffic, highlighting the need for better protection for small businesses in the face of Meta's widespread web crawling practices.