Build a Reproducible AI Tool Benchmark in Python

Most AI tools appear impressive during polished demonstrations, but determining their reliability, speed, and affordability for your specific needs proves challenging. A universal leaderboard falls short in addressing these concerns. Your team may prioritize extracting fields from invoices, explaining code, drafting support replies, locating citations, or generating study notes, each of which presents unique failure modes. Consequently, the top-performing model for one task might not be suitable for another.

This tutorial provides a method for creating a small, reproducible Python benchmark to compare AI tools across four practical dimensions: task-specific answer quality, latency, consistency across repeated runs, and estimated usage cost. The benchmark harness utilizes only Python's standard library. Each candidate is connected through a simple adapter, the same test cases are executed multiple times, and both detailed JSON and a summary CSV file are generated as outputs.

The rationale behind building your own AI benchmark lies in the fact that general-purpose benchmarks, while useful for discovering candidates, typically evaluate broad capabilities under controlled conditions. Production decisions, however, are more focused. For instance, a team selecting an AI research assistant may require answers with clickable sources, a coding team may prioritize correct patches and low latency, and an educational institution may prioritize accurate, age-appropriate, and verifiable explanations.

To create an effective benchmark, it should possess five key properties: representativeness, repeatability, multi-dimensionality, auditability, and brevity. The cases used in the benchmark should resemble real-world prompts, documents, and constraints. The inputs and settings should remain consistent for every candidate. Quality should be assessed alongside speed and cost.

The raw outputs must be preserved, not just a single composite score, and the suite should be easily rerun after any changes to the model, prompt, or pricing.

Step 1: Define the decision before determining the metrics. Clearly outline the criteria that would lead to adopting or rejecting a candidate. A simple decision rule might be: Minimum quality score of 0.85, median latency under 4 seconds, zero serious factual errors, and an estimated cost per 1,000 tasks under $10. By establishing these criteria beforehand, you can avoid changing the rules until your preferred tool emerges as the winner.

Initially, filter out any candidates that fail to meet these hard requirements, and then rank the remaining options.

Step 2: Develop task-specific cases. In the provided example, each case includes a stable identifier, a prompt, terms that a minimally acceptable answer should contain, terms that should not appear, and whether a clickable source is required. This keyword coverage serves as a simple smoke test, rather than a comprehensive measure of correctness.

For high-stakes or open-ended tasks, consider incorporating a human rubric or a carefully validated evaluator. The @dataclass (frozen = True) class Case is defined with attributes for a stable identifier, prompt, required terms, forbidden terms, and a boolean flag for requiring a source.

The CASES variable contains twenty to fifty carefully selected cases, encompassing normal cases, edge cases, and adversarial cases. Ensure to remove any private customer information before sharing the cases with external services.

Step 3: Standardize the response format for all candidates. Different providers may return usage data in various formats. To address this, a small adapter is used to normalize the responses. The @dataclass (frozen = True) class ModelResponse is defined with attributes for the generated text, input tokens, and output tokens. Another @dataclass (frozen = True) class Candidate is defined with attributes for the candidate's name, the callable to execute the model, input and output token costs in USD per million tokens.

Adapters should accept a prompt and return the text along with the provider-reported token counts. The specifics of the provider, such as whether the response comes from a hosted API, a local model, or a workflow tool, should be encapsulated within the adapter. Record the exact model version, system prompt, temperature, tool settings, region, and date outside the code to ensure reproducibility later.

Step 4: Score quality transparently. The quality_score function awards 75% for required-term coverage and 25% for satisfying the source requirement. Forbidden terms incur a penalty. The function takes the case and the generated text as inputs and returns a rounded score between 0.0 and 1.0. This transparent rubric can be easily inspected, modified, and explained as needed.