Your AI Agent Is Not a Chatbot Anymore. It's a Distributed System.

AI agents have evolved beyond being simple chatbots. When an AI agent starts calling APIs, maintaining state, waiting for events, and interacting with the real world, the engineering challenges significantly increase. Initially, people used the analogy of a chatbot with a few extra features to understand AI agents. This mental model was sufficient for early experiments.

However, the real problem arises when the agent becomes useful. Allowing an agent to update a CRM record, create a support ticket, approve a transaction, check inventory, change infrastructure, send emails, process refunds, or trigger workflows alters the architecture in a manner that is not immediately obvious. The interface may still resemble a conversation, but underneath, the system is no longer merely generating responses.

It coordinates state changes across multiple systems. This shift introduces numerous challenges that are not present when the agent is just a chatbot. These challenges include dealing with databases, APIs, authentication, network failures, concurrency, retries, stale data, partial execution, long-running workflows, permissions, and recovery mechanisms.

These issues become even more critical as the model's capabilities grow. While a model that can only generate text has a limited impact when something goes wrong, a model capable of operating across enterprise systems carries a much larger risk.

The architecture of production AI agents differs from that of chatbots with tools. It is not just a chatbot with additional functionalities. Instead, it is a software process that includes a probabilistic decision-making component operating within a distributed system. Once you view AI agents in this light, many problems initially perceived as AI issues start to resemble familiar software engineering challenges.

A common demonstration of an agent involves a user requesting something, the model deciding what action to take, a tool being called, a result returned, and the model producing a final response. This sequence may look like this: User ↓ Agent ↓ Tool ↓ Result ↓ Agent ↓ Response. While this architecture is acceptable for a prototype, it does not fully capture the complexities of production software.

Issues such as tool response times, network disconnections, concurrent requests, outdated data, process crashes, and undetermined operation success become apparent. For instance, consider a scenario where an agent is handling a customer's refund. The agent instructs the payment service to process a $500 refund. However, the network connection between the payment service and the agent is lost before the response is received.

From the agent's perspective, the request timed out, while the payment system believes the refund was successful. Two independent systems now have conflicting versions of reality. This situation is not a prompt engineering problem but a distributed systems problem. The model's intelligence should be separated from the guarantees provided by the software infrastructure.

While an LLM can reason about what should happen next and adapt to unexpected results, it should not be responsible for ensuring deterministic execution of software. If an agent decides a customer should receive a refund, it should not be responsible for guaranteeing the refund occurs only once. Similarly, if an agent decides to modify an account, it should not have to determine whether it has authorization to make the change.

If an agent believes a previous operation failed, it should not have to speculate about whether the operation was actually successful. These concerns are related to infrastructure rather than the model's decision-making capabilities. By separating the reasoning and execution layers, an agent can maintain flexibility and probabilistic reasoning while the surrounding system remains deterministic wherever possible.

This separation becomes increasingly important as agents are deployed in higher-consequence environments. It is also crucial to avoid viewing tool calls as ordinary function calls. A tool call might involve multiple network boundaries and complex processes, such as authentication, data validation, and error handling. Treating a tool call similarly to a function call can lead to misunderstandings about the integration's complexity.