Your Drupal robots.txt Changes Aren't Showing Up: Here's Why
Edited robots.txt in Drupal but the live file still shows old content? Composer Scaffold, static files, and CDN caches are usually why — here's the fix.
Your Drupal robots.txt file isn't updating as expected, even after clearing cache and running cron. This issue can stem from several factors. Firstly, if your project utilizes the Composer scaffold, it may regenerate the robots.txt file on each deployment, overwriting any manual changes. To resolve this, modify composer.json by adding `"file-mapping"` for the robots.txt file and setting its value to `false`.
This instructs the scaffold to skip the file during installation and updates. Additionally, ensure that the robots.txt file itself is deleted from the webroot after the change, to prevent any existing static file from taking precedence. Other potential causes include static files or directories in the webroot that override the module, Drupal's cache not being cleared correctly, hosting platform caching layers, or CDN and reverse proxy caching.
By systematically checking each of these aspects—composer.json configuration, existence of static files, cache clearing, response header verification, platform-specific caching, CDN purge, and web server configuration—you can identify and address the root cause, allowing the robotstxt module to serve the robots.txt file dynamically.
Written by urgent.news from HackerNoon's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
