How a Website Becomes Citable by Search Engines and AI Assistants
TL;DR A citable website is a site whose pages a crawler can fetch without a login, read as text in the HTML it receives, attribute to a named organisation and cross-check against a structured data block. The first pass reads what the crawler actually receives: robots.txt rules, CDN and hosting rules, and server-rendered HTML with the text inside it. OpenAI documents four user agents with separate…
A citable website is one whose pages can be accessed by crawlers without requiring a login, and the content can be read as plain text within the HTML received. This includes adhering to robots.txt rules, CDN and hosting guidelines, and having server-rendered HTML that includes the textual content. OpenAI has identified four user agents, each with distinct roles: OAI-SearchBot for ChatGPT search, GPTBot for training purposes, ChatGPT-User for user-initiated actions, and OAI-AdsBot for ad landing pages.
Google's AI features utilize the same index as regular search, and it requires allowed crawling, discoverability through internal links, textual content, and structured data that accurately reflects the visible content. Structured data should be in JSON-LD format and represent what users can see on the page. The llms.txt file is a proposed markdown index at a site's root or path, containing an H1, summary blockquote, and links to markdown copies.
This file is situated alongside robots.txt and sitemaps. The four-pass check involves fetching, reading, identifying, and structuring a page; any failure indicates specific build issues. To achieve citation readiness, a page must load, contain readable text, and provide information about its author and subject matter. Additionally, companies need to familiarize themselves with the documentation of the systems that perform the quoting, as each has its own set of rules regarding crawling and content inclusion.
Both obtaining a crawlable page and adhering to specific rules require clear and honest file and tag structures.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
