The cheapest way to stop your AI product from regressing

Generative AI systems can perform well in conventional software testing but may exhibit deteriorating behavior in real-world applications. This issue is exacerbated by the open-ended nature of their outputs, which can vary significantly even for the same input. A common mistake is relying on informal testing with a few examples, which may not reveal the system's true limitations.

To mitigate these risks, a small, carefully curated collection of real user inputs known as a golden dataset is crucial. This dataset should contain a representative sample of user interactions, including ambiguous requests, unsupported languages, and missing information. By running these examples whenever the model or system is updated, potential regressions can be identified and addressed before they affect users.

The golden dataset should not replace other monitoring methods like production monitoring or user feedback, but rather serve as an additional layer of quality assurance. It should be constructed using real production examples, synthetic examples can be used initially, but should be gradually replaced with real interactions. The goal is to capture a recognisable sample of how users actually engage with the product, not to create a perfect benchmark.

When evaluating the AI's performance, focus on the outcome or desired behavior rather than requiring an exact answer. This approach accommodates the variability in human language and ensures the system meets its functional requirements. The dataset should be kept small enough for easy inspection and versioned alongside the product to track changes and improvements.