Your Java LLM agent works in the demo. Friday night in production is another story.

On Friday night, an LLM agent designed to handle customer refunds demonstrated a critical flaw when it began returning 503 errors, causing request timeouts and accumulating in the queue. Meanwhile, the same model started wrapping JSON responses in Markdown code fences, which downstream services were unable to parse. This created significant issues, especially when a customer requested a €4,000 refund, as the agent proceeded without the proper safeguards in place.

On Monday morning, the team was left with two major questions: what were the costs of the weekend, and what actions had the agent taken? The problems faced during this incident were not related to the intelligence of the model but rather the lack of proper rules and controls governing its behavior in production. ARA (Agent Runtime Architecture) is an open-source Java runtime for LLM agents that aims to address these issues by providing a structured and deterministic approach to agent execution.

ARA is built on LangChain4j and offers features such as output contracts, human approval, provider failover, spend caps, and execution traces. It operates entirely offline for testing purposes and is available on Maven Central as io.github.xmor:ara-runtime:1.0.3. The library allows developers to define deterministic rules and validations for agent outputs and actions, ensuring that the model's probabilistic decisions are followed by the runtime's rules.

By implementing ARA, developers can prevent malformed outputs, human oversight for critical actions, provider downtimes, cost management, and decision reconstruction during failures. The key takeaway is that while LLMs can help decide what to do, they should not be solely responsible for the rules governing the execution of those decisions.