Making a Next.js Site Readable by AI Agents: llms.txt, a Markdown Mirror and Accept: text/markdown

As more users rely on AI agents to access websites, rather than directly viewing HTML, the need for AI-friendly content becomes increasingly important. To address this, a Next.js site was developed with a Markdown mirror and specific headers to ensure AI agents can easily read its content.

The site, AskedTheAI, is a shopping guide that queries four AI models for the same buying question and has a human review the selections on Amazon. To make the site more AI-friendly, four key elements were implemented:

1. A Markdown copy of each page, generated from the generated content during the build process. This simplifies the maintenance process, as the Markdown is derived from the static HTML pages during the build.

2. Content negotiation through an Accept header. When an AI agent requests the page, it can specify a preference for Markdown content, and the server will respond with the corresponding Markdown file instead of the HTML. This enables AI agents to consume the content in a format that is more efficient for them.

3. Mapping the Markdown files through llms.txt and llms-full.txt files. These files serve as a directory of all routes with Markdown mirrors, allowing search engines like Bing and Yandex to quickly learn about updates to the site.

4. The site is built using Next.js 16 (App Router, fully static) and deployed on Vercel. The entire setup is derived from the source code available in the project's repository.

The process involves generating the Markdown mirror from the already built static HTML pages, skipping unwanted elements like scripts, styles, and iframes. Front matter is added to each Markdown file, providing essential information about the page, including its title, URL, description, last updated date, and source. The middleware then handles the Accept header, allowing AI agents to request the Markdown version of a page when requested.