The More Powerful the AI, the More the Architecture Matters

The architecture of AI systems plays a crucial role in determining their effectiveness, according to a recent experiment building an LLM-powered support agent with deterministic boundaries. The experiment involved creating a support agent that could determine whether a customer was owed a refund without actually providing one. The AI system read the customer's message and found relevant documents.

The software then decided eligibility, owned risk tiers, scoped lookups to the logged-in user, and placed expensive tasks in a queue for human intervention. The entire system ran without any API key.

The key takeaway from the experiment is that the boundaries set for the AI system were crucial to its success. These boundaries included the gaps the experiment hadn't solved, the reasons why the difference mattered, and a set of tests that would fail if the system stopped being true. One of the most important boundaries was an allowlist that prevented the business rules from referencing any packages other than itself and the JDK.

This allowed the system to maintain its deterministic behavior and prevented any unexpected dependencies from creeping in.

Another important aspect of the architecture was the gate that determined whether the AI or a human would handle a particular task. The gate was designed to only allow tasks with a low risk tier to be handled by the AI, while medium to high risk tasks required human approval, and very high risk tasks were only proposed and required a person to execute them. This gate helped ensure that the system remained under human supervision and did not become too autonomous.

The experiment also highlighted the importance of having a well-defined architecture that prevents the system from slowly decaying into something that no longer meets business requirements. By having an allowlist and a deterministic gate, the system was able to maintain its focus on business questions and avoid becoming overwhelmed by technical debt. The experiment did not include a full implementation with real customers, but it did provide a framework for building a more robust and reliable AI system in the future.