AI Agents Don't Have a Timeout Problem. They Have a Waiting Problem.

AI agents are not plagued by a timeout issue; rather, they face a waiting problem. The real challenge lies in teaching these agents how to wait effectively without losing state, consuming resources, repeating work, or disrupting the workflow. Traditional AI agent architectures are built around a user request. The agent processes the request, calls a tool, receives a response, and then thinks again.

This approach works well when all components in the workflow operate swiftly. However, when an agent operates in the real world, it encounters delays. Some tools may take thirty seconds, others may take two minutes, and certain services might require human intervention or long processing times. This waiting problem is easy to conceptualize but challenging to implement.

Most AI agent architectures are designed around a request-response model. Users submit a request, an agent thinks about it, calls a tool, receives the result, and then generates a final response. This model functions admirably for short-lived operations, where everything is completed within an HTTP request's timeframe. However, when the agent's workflow extends beyond this brief window, issues arise.

For instance, an agent waiting for human approval, payment confirmation, or deployment completion might seem idle. Yet, it is actively engaged in the workflow, not stalled.

The problem with treating every agent task as an HTTP request becomes apparent when developers attempt to apply this pattern to lengthy workflows. Consider an onboarding agent for a new enterprise customer. This agent must create an account, verify documents, conduct compliance checks, request administrative approval, provision access, notify the customer, and wait for confirmation that the customer has completed the setup.

Such a workflow may span an hour, a day, or even several days, depending on various external factors. Keeping the HTTP request open for such extended periods is impractical and risky, as it could be disrupted by various system failures or network issues.

Recognizing the distinct nature of waiting from failure is crucial. A timeout indicates that a request did not receive a response within a specific period, but it does not necessarily reveal what transpired on the other end. In distributed systems, states such as SUCCESS, FAILED, PENDING, or UNKNOWN exist. However, a timeout should not automatically classify the operation as FAILED.

Instead, the system should attempt to ascertain the operation's status through mechanisms like idempotency keys or transaction identifiers, which can often confirm that the operation was completed successfully despite the timeout.

Incorporating waiting as a first-class state in agent workflows is essential. Instead of executing loops like 'while not approved: time.sleep(10),' workflows should persist a state like 'WAITING_FOR_APPROVAL' with a timestamp of when the waiting began. This approach allows the agent to maintain context and react appropriately to changes in the workflow's status, rather than blindly retrying operations that have already succeeded.

By understanding waiting as a critical component of workflow management, AI agents can operate more reliably and efficiently in real-world scenarios.