A quick site hardening pass: email DNS, AI-crawler rules and security headers
Three things go quietly wrong on a lot of small sites: mail that fails authentication, a robots.txt that says something different from what you meant, and missing security headers. None of them break the site, so nobody notices. Here's a short pass you can run on your own domain with curl (and jq for the DNS part). No sign-ups, no installs beyond those two. 1. Email DNS: SPF, DKIM, DMARC (and MX)…
Three key areas of website security often get overlooked: email DNS settings, AI crawler rules in the robots.txt file, and security headers in HTTP responses. Each of these can be addressed with relatively simple steps that are easy to implement and maintain.
Firstly, email DNS should include SPF, DKIM, DMARC, and MX records. A shell function can be used to query Cloudflare's DNS-over-HTTPS resolver to retrieve these records. Once retrieved, ensure that the MX record points to a fully secured domain and that the SPF and DMARC records are properly configured with tight restrictions. The robots.txt file should also be reviewed for any rules related to AI crawlers such as OpenAI's GPTBot and Anthropic's ClaudeBot.
These rules should be clear and specific, with shared disallow rules included in named groups to avoid confusion. Lastly, HTTP security headers such as Strict-Transport-Security, X-Content-Type-Options, Referrer-Policy, and X-Frame-Options should be present and set to recommended values. These headers help to secure the site by enforcing HTTPS, preventing content type sniffing, controlling referrer information, and mitigating clickjacking attacks. Regularly checking and updating these settings can help maintain a secure website environment.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
