I measured what 300 websites actually serve to AI agents

A researcher measured the content served to AI agents across 300 public websites. The study found that cryptographically signed AI agents were treated differently from unsigned ones, but the differences were not favorable. Hidden machine-only text was discovered on one page, belonging to firecrawl.dev. The cost of a machine reader per page averaged 2592 tokens.

Half of this cost was not content-related, with navigation, headers, and footers contributing to the remainder. Seven out of 300 sites served markdown variants to machine readers, delivering the same content at a median cost of 962 tokens. Six of these markdown variants were missing Vary: Accept, a header that informs caches about the content's dependency on the client's request.

Three of these seven also omitted real body content from the markdown version. The researcher encountered issues in measuring identity effects on live pages, primarily due to the pages changing during the measurement process. To address this, they focused on interior content pages, fetched unsigned requests multiple times, and defined a stable core as the blocks identical across these requests.

This approach reduced baseline instability from 27% to under 4%. The researcher also corrected previous errors in cost calculations and markdown content loss figures, and tightened the rules for detecting content loss in markdown conversions. The tool used for this study, called "content-parity," is an open-source solution that fetches pages under various identities and provides a detailed comparison of the content received by each.