Blunt Banhammer Blocks Basic Bots Ballistically

A hobby website owner reduced bot traffic to their site by over 80% using a simple trap. The site, OpenBenches.org, transferred more than 2.5TB of data to visitors. The author created B6, a tool inspired by Jeff Starr's Blackhole, to eradicate unwanted bot traffic. B6 works by utilizing robots.txt and .htaccess files to block bots from accessing the site.

Bots typically check for updates every 24 hours, so the author either waits a day or manually submits the changes to major crawlers. The site's .htaccess file contains instructions for how the server should process requests, and the author modified it to send an error code and a readable message to blocked IP addresses. A honeypot directory contains an index.php file that records the visitor's IP address and User Agent when the honeypot page is visited.

Web crawlers follow links on a page, so the author added a hidden link to the honeypot on every page, designed to be invisible to normal users but clickable by bots. The honeypot link is hidden from normal viewers using CSS and impossible to click due to zero-width joiner text. Bots might ignore robots.txt, so the author added an allow list exemption for identified bots.

The blocked IP addresses have timestamps associated with them, allowing the author to periodically unblock older entries. The author has blocked over 400 IP addresses since implementing B6, resulting in a significant reduction in traffic and no complaints from users.