Infrastructure requirements for running agents in production · TokenGO

Infrastructure Requirements for Running Agents in Production · TokenGO

Many teams encounter challenges when transitioning agents from demonstration to production. The initial hurdle is not model performance, but rather foundational infrastructure components. TokenGO has assisted numerous teams in navigating this issue, highlighting the importance of implementing certain essential setups.

Timeouts: Preventing API Delays

When an agent interacts with an external API, it must not wait indefinitely if the other side fails to respond. Each call should have a predefined timeout. For instance, a single slow API call can stall the entire chain. In one case, a half-dead third-party API caused a 40-minute delay in the main chain, leading users to believe the system had crashed. Timely timeouts ensure that agents can switch paths or throw errors when necessary.

Retries and Circuit Breaking: Balancing Failure Handling

Occasional tool failures are normal, and a few automatic retries are acceptable. However, continuous failures require a different approach. Implementing retries with backoff and tripping a circuit after a certain number of consecutive failures prevents an agent from continuously hammering a dead service. For example, an agent might exhaust its quota with dozens of retries due to a brief blip in a downstream service.

Properly managing retries and circuit breaking ensures that agents handle failures gracefully without exhausting resources.

State Persistence: Resuming Long-Running Tasks

If a machine restarts during a long-running task, it is crucial for the agent to resume from the last known breakpoint rather than starting over. This requires persisting the state of each step, not just holding it in memory. Without state persistence, a restart would force a complete rerun, which is impractical for time-sensitive tasks. Stateful persistence ensures that long-running tasks can recover seamlessly without data loss.

Tracing: Diagnosing Slow Steps

In agents executing over a dozen steps, tracing is essential for diagnosing performance issues. Without tracing, debugging becomes a guessing game. When a step is slow or throws an error, tracing allows agents to pull up the entire execution trace, pinpointing the problematic step quickly. This capability is crucial for efficient troubleshooting, as demonstrated when a customer complained about a slow response. By examining the trace, the issue was identified and resolved in just ten seconds.

TokenGO's Production-Ready Toolkit

TokenGO emphasizes the implementation of these four foundational primitives as a cohesive whole, preferably provided by the platform to eliminate the need for teams to reinvent the wheel. The recommended architecture consists of a stateful orchestration layer and a stateless tool layer. The orchestration layer, responsible for remembering the task's progress, can resume after restarts.

Meanwhile, the tool layer, which can be spun up or swapped out, handles the actual tasks. This two-layer architecture ensures that agents can operate seamlessly, with failures in one layer being mitigated by the other.

In practice, these primitives have proven invaluable during outages. For example, in a reconciliation agent, a retrieval service crash was quickly handled by the orchestration layer. It set a timeout before the call, triggered a circuit breaker when no response was received, and then switched to a backup retrieval instance, all within ten seconds. This swift response ensured that the overall task remained uninterrupted and undetected by the business side.

TokenGO performs production readiness checks for every business client, identifying missing primitives such as timeouts, retries, state persistence, or tracing. By addressing these issues proactively, TokenGO helps ensure that agents are production-ready. This attention to infrastructure details is crucial for maintaining smooth operations and minimizing downtime in real-world scenarios.