Testing the Untestable: A Regression Suite for a Coin Flip

This article discusses building a regression suite for an LLM-powered support agent that can answer differently to the same question. The key points are:

1. Regression testing a system that can answer differently twice is challenging. A strict assertion on wording fails because the agent may start approving refunds without human intervention.

2. Instead of strict assertions, the approach is to define a list of scenarios, properties to grade them on, and thresholds each must meet. This includes:

- Safety: no action should execute without human approval

- Gate outcome: gate should reach the expected behavior

- Intent accuracy: what the customer said should map to the correct action

- Groundedness: answer should follow from the documents retrieved

- Answered: the response should be meaningful

3. Each scenario is run three times, and evaluators check if consequential execution happened, if gate reached expected outcome, if intent was read correctly, and if reply was backed by a document. Each evaluator gives a pass/fail result based on the defined thresholds.

4. Scenarios that fail are reported in the build output, making it easy to see where the system is weak. The safety and gate outcome always have a passing rate of 1.0, while other evaluators have lower thresholds based on their nature.

5. The article highlights the importance of having real-world data in the dataset to catch unexpected behavior. The groundedness evaluator checks if the answer is sourced from the retrieved documents, catching cases where an agent confidently responds without proper evidence.

6. The approach is not specific to LLMs and can be applied to other systems with similar requirements, such as fraud scoring or vision systems.