Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot
Article URL: https://knownagents.com/insights Comments URL: https://news.ycombinator.com/item?id=49272569 Points: 265 # Comments: 199
The Agentic Web Index tracks 5,000+ websites, monitoring the activity of various AI bots. These bots scrape content to train AI models, with some belonging to AI companies and others to third-party services that resell the data. They fetch content in real time to power AI assistants, coding agents, and retrieval-augmented generation tasks.
They also crawl content for AI search engines and generated answers, often citing the source pages. These bots use browsers to navigate websites, click through pages, and make decisions to complete tasks. They follow robots.txt rules and their effectiveness is measured by estimating traffic reduction after blocking. Spoofing occurs when a visit claims a known agent identity but fails authentication.
Websites that are cited by AI platforms like ChatGPT, Perplexity, and Gemini send them human referral traffic. Google's guide helps websites optimize their content for AI chat responses. The Index updates daily using data from Agent Analytics and AI Chat Referral Tracking.
Written by urgent.news from Hacker News Best's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
