Building an honest scanner for the agentic web

The author of this article aimed to create an honest scanner for the agentic web. Many AI readiness scores for websites are always red and do not provide clear information about missing items. The author differentiated between sites that have not published an emerging protocol and those that have published one but it does not work.

The first situation is usually an observation, while the second is a genuine defect that needs attention. The author wanted to measure honestly and help implement the findings. The scanner is built in a container organized into six categories: Discoverability, Content, Access, Capabilities, Commerce, and Trust. It detects, validates, and reports on 23 technologies, with 7 watched but not scored.

The scanner emits the same shape for each technology: tech Present boolean, tech Valid boolean, and tech Defect boolean. The scanner only emits data for technologies that are present and defect-free, avoiding unnecessary validation and reporting. The author also identified two kinds of signals: page-level signals that vary per URL and site-wide signals that are one fact per host.

The scanner fetches well-known files once per URL and once per host, avoiding unnecessary fetch calls for unchanged files. The scanner's type system prevents false positives caused by timeouts or other issues. The scanner uses two registries: defect-catalog and tech, which provide information about what can go wrong with a technology and what the technology is, respectively.

These registries are used to generate the scanner's output and ensure consistency between the scanner, documentation, and reports. The author also built three Claude Code skills to maintain the scanner's accuracy, as emerging standards and proposals change frequently.