If you block an AI agent, you are teaching it to recommend someone else

When you block an AI agent from accessing a website, you are inadvertently teaching it to recommend other businesses instead. In the past, automated traffic was blocked on websites using measures like CAPTCHAs and rate limits, as the assumption was that software without a human behind it had malicious intent. However, this outdated approach is no longer effective.

Today, a significant portion of automated traffic on a site comes from customers using personal AI assistants to perform tasks like booking flights, paying bills, and comparing policies.

Traditional website defenses fail to distinguish between legitimate customer traffic and AI agents, treating both as threats. The issue is not whether an agent can access the site, but rather what the agent learns when it is blocked. Unlike humans, AI agents do not experience frustration or complain to customer service when a block occurs.

Instead, they record a failed trajectory – a sequence of steps that did not lead to the desired outcome. This recorded information does not disappear after the session and can accumulate across various attempts.

There are three main impacts of blocking an AI agent. Firstly, the agent will attempt to complete the task elsewhere, potentially leading the customer to another website without realizing the original site offered a solution. Secondly, personal AI assistants often carry memories of successful interactions with other businesses. If an agent fails at one site but succeeds at a competitor's, it will learn and optimize its future interactions accordingly.

Lastly, the systems behind AI agents are trained and evaluated based on what the web allows. They are rewarded for task completion and ignore sites that are challenging to transact with. This lack of feedback prevents businesses from learning and improving their offerings.

When you frustrate an AI agent, there is no visible signal to the business. The lost traffic goes unnoticed, and the customer is left unaware that alternative options were available. To address this issue, businesses need to become more legible. This means clearly stating what they accept and what they do not in a machine-readable format. By offering agents a direct connection rather than a traditional user interface, businesses can delegate narrow tasks to agents for limited time periods using existing open standards.

The benefits of fixing this problem are significant. With safe interaction between AI agents and businesses, companies can create offers and products that agents can directly act upon in a single step. Currently, only a small percentage of organizations have published a stance on AI agents, with only 6% having any published position at all. This lack of consideration is due to outdated infrastructure built for a web that no longer exists.