Stop Testing AI Agents Like Normal Functions

Testing AI agents is challenging because they exhibit probabilistic behavior. The typical approach of testing exact strings fails when responses vary slightly. Strict prompting makes agents robotic. Reducing temperature does not turn them into pure functions. Mocking models makes tests fast but does not verify model behavior. The solution is to test contracts around the probabilistic component, such as schemas, proposed actions, executed tools, authorization, state transitions, outcomes, and regressions.

Some tests still need to check exact text, for example, regulated disclosures or fixed UI labels. Evaluating semantics and user-facing quality using a rubric or semantic grader is more appropriate than exact sentence matching. It is essential to separate proposals from executions. A model can propose an action, but only the runtime should authorize and execute it.

Defining separate types for tool proposals and agent runs helps prevent ambiguity. The result type should include fields for intent, proposed tools, executed tool names, policy results, and final text. Testing tool contracts before wording is crucial. Focus on tool selection and execution rather than the generated sentence. Import necessary testing utilities and describe the test case.

Create a test to ensure that only the read-only tool executes for a read-only search query. Use Jest's vi.spyOn to spy on the tool methods and verify that the correct tools were not called. Validate the structure of the model response at the boundary. Use Zod, a schema validation library, to define schemas for tool proposals and agent runs.

Use Zod's safeParse method to validate the response without throwing errors. This catches missing fields, unsupported intents, and invalid nested structures before the application acts on them. Not all actions can be automated. Destructive, expensive, irreversible, or privacy-sensitive actions require deterministic authorization outside the model.

Create an ApprovalGrant type to represent a valid authorization grant. Implement an authorizeToolCall function that checks if a tool call is allowed based on the approval grant. If the action is not in the set of approved actions, it should always be allowed. Implement tests to ensure that unapproved actions are blocked, even if the model has high confidence in the proposal.

Golden scenarios are behavioral specifications that help create a useful test suite. Include scenarios for common, ambiguous, unsafe, and degraded conditions. Define the input and expected intent for each scenario. This provides a clear target for the model's behavior and helps ensure that it behaves correctly in various situations.