Can AI Agents Actually Use Your Website? I Built a Tool to Find Out

AI agents are becoming increasingly capable of interacting with websites, but a fundamental question remains: Can an AI agent truly use your website? To explore this, I created a tool called Agentic Web Check, which tests a website in a real browser environment. This approach goes beyond merely examining HTML, metadata, accessibility, or AI-specific interfaces. Instead, it attempts to use the site in a practical manner, checking tasks such as locating contact details, privacy policies, and help pages.

The most crucial aspect of Agentic Web Check is its programmatic verification of whether each task is successful. The tool does not simply pass or fail; it assigns one of four outcomes - PASS, FAIL, BLOCKED, or INCONCLUSIVE - based on assertions about the browser's state after the task is attempted. To ensure safety, there is an additional layer that scans for potential issues like hidden instructions for AI systems, unsafe forms, authentication limits, downloads, and actions without proper safeguards.

During testing, one fixture scored 81 on perception checks but encountered BLOCKED status for all three browser tasks due to a consent banner lacking a properly named close control. This is a controlled, intentionally flawed fixture to demonstrate the distinction between static audits and actual agent operation. Currently in its early stages (v0.1.0), the project is Chromium-centric and lacks a comprehensive site crawl or public benchmark. Some thresholds are set as generic defaults rather than data-driven constants.

The project is open-source and available at https://github.com/ericovirgy/agentic-web-check. I am particularly interested in feedback from those working on browser agents, Playwright, accessibility, WebMCP, and other agentic web tools. What additional metrics would you suggest?