How Do You Actually Test an AI System? A Layered Strategy From Five Tools I Built

Testing AI systems presents unique challenges not faced with traditional software. Two main issues arise: nondeterminism, where the same input can yield different outputs, and the lack of a single correct answer in many scenarios. Traditional testing methods focus on correctness, but AI testing requires a different approach, focusing on whether the system has regressed and can be trusted for specific outputs.

To address these challenges, the author built a layered testing strategy divided into five open-source projects:

1. Layer 1: Evals as a build artifact - This layer treats quality metrics as part of the CI process, similar to how code quality is measured. The evalgate project creates a declarative evaluation suite that runs on every pull request, comparing the current quality score against a baseline. It provides various scorers to measure both strict and fuzzy aspects of the AI system. By running on a deterministic mock provider, the process works offline and doesn't rely on any API keys.

2. Layer 2: The modality text evals cannot see - Text evaluations are insufficient for certain types of failures, such as those in voice-based agents. The voiceeval tool addresses this by capturing the actual spoken words from the caller, allowing for detection of misheard numbers or other conversational issues that text-based evaluations would miss.

3. Layer 3: Record and replay - To evaluate AI behavior over time, the replay tool, Tracecase, records agent interactions and compares them against previous runs. This helps identify regressions and safety issues that may have been missed in earlier evaluations. The tool generates a shouldFail signal that can be integrated into the CI process.

4. Layer 4: Trust individual answers with receipts - While regression gates help maintain overall system quality, they don't guarantee the trustworthiness of individual answers at a given moment. The answerproof project adds cryptographic signatures to generated answers, providing a tamper-evident receipt that documents the sources used, permissions, model parameters, and other relevant information. This allows others to verify the answer's integrity independently.

5. Layer 5: Observability over what agents actually did - The final layer focuses on monitoring AI system behavior in production. Agentrace analyzes agent transcripts and session records, providing visibility into what the agents actually did. This helps identify cases where the system's behavior may not be trustworthy, even if the code passes all previous tests.

By stacking these layers, the testing strategy provides a comprehensive approach to ensuring AI systems maintain quality and trustworthiness over time. Each layer builds upon the previous one, addressing different aspects of AI system testing and providing objective, comparative signals to guide the development process.