Why Your Static Site’s lastmod Is a Lie (And How CI/CD Shallow Clones Secretly Hurt Your SEO)

If you build static websites using Next.js, Astro, or 11ty, your sitemap.xml might be misleading search engines about your site's last modification dates. This article explores why this happens and how continuous integration/continuous deployment (CI/CD) shallow clones can exacerbate the problem.

In a CI/CD pipeline, a shallow clone is typically performed using the command `git clone --depth 1`. This means that Git only keeps track of the latest commit and does not maintain a complete history of the repository. As a result, every file in the repository is timestamped with the current time when the build is triggered, regardless of when the files were actually modified. This leads to inflated last modified dates in the sitemap.xml file.

Search engines like Google and Bingbot are not fooled by these false timestamps. When they see all URLs on a domain marked as modified on the same day, they interpret this as artificial freshness manipulation and may penalize the site by dropping the lastmod metadata from its priority queue. This can cause your crawl budget to be wasted, as search engines will ignore the lastmod headers entirely, leading to slower indexing of fresh articles and repetitive crawling of unchanged pages.

Some developers attempt to resolve this issue by querying Git directly for the last commit date of each file. However, this approach also has limitations. When using a shallow clone, Git only retains the latest commit history, and any files not modified in that commit are treated as having no commit history. Additionally, adding `fetch-depth: 0` to the GitHub Action to fetch the entire commit history can significantly increase build times and costs.

The article also discusses a subtle bug related to `git status --porcelain`, which can misinterpret dirty files (i.e., files with uncommitted changes) and mistakenly use their timestamps instead of the committed dates. This bug can further complicate the accurate representation of last modification dates in the sitemap.xml file.

In summary, relying on static timestamp values in your sitemap.xml can lead to SEO issues and wasted crawl budgets. It's essential to use more accurate methods for determining last modification dates, such as Git log or considering the limitations and potential pitfalls of each approach.