Which Malaysian websites block AI crawlers? I checked 100
First published on yappatyih.com . TL;DR An AI crawler block is a rule in a website's robots.txt file that tells one named bot, such as OpenAI's GPTBot, not to fetch the site's pages. On 20 October 2026 I read the robots.txt of 100 major Malaysian websites in ten sectors, from banks and telcos to news, government, shopping and universities. 13 of them block at least one AI crawler. None blocks…
On 20 October 2026, a report revealed that 13 out of 100 major Malaysian websites blocked at least one AI crawler. OpenAI's GPTBot was the most blocked crawler, with 9 sites blocking it, followed by CCBot and Bytespider, each blocked by 8 sites. None of the sites blocked all 17 AI crawlers they encountered. Only 4 sites blocked Google-Extended, the crawler used by Google for indexing purposes.
News sites blocked AI crawlers the most, with 5 out of the 15 news sites blocking at least one. Banks, telcos, travel sites, and universities did not block any AI crawlers in the sites they checked. Most sites (74) did not block any AI crawlers or had no robots.txt file at all.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
