New Model Dropped? Run Your Own Git History Through It First
The release notes say it's faster. The launch thread says it beats everything. Three people I follow have already switched. And yet, every time I've switched on that basis alone, I've quietly switched back two weeks later after the model mangled a migration script or confidently explained a bug that didn't exist. The gap is simple: public benchmarks measure what benchmarks measure. My daily work…
When a model is introduced, I test it myself before adopting it. I don't rely on rankings or public benchmarks, which only measure certain aspects of a model's performance. Instead, I create a ritual to evaluate the model with my own work. This post outlines the steps I take to decide whether a model is suitable for my needs.
First, I ask three questions to determine if a model is good enough for my use case. However, these questions are not sufficient on their own. The first question is whether the model performs well across different prompt shapes. The second question is whether I can afford to be careless with the model, considering factors like cost and performance. Finally, the third question is how the model fails, specifically whether it can produce uncertain but correct results rather than confidently wrong answers.
To test these questions, I mine my own history for test cases. I keep all the prompts in one folder, with each case in a separate YAML file. The cases cover various scenarios, such as untangling a Bash script with a quoting bug, adding type hints to a function without changing its behavior, diagnosing a traceback, and translating a jQuery snippet to vanilla JavaScript.
Each file contains the prompt and checks that need to be performed. The checks are divided into two types: mechanical and human judgment. Mechanical checks involve verifying that the model produces correct output, such as parsing the script, compiling the code, or ensuring the output has an exit code of zero. Human judgment checks rely on the evaluator's expertise to determine correctness, such as identifying the quoting issue in the Bash script or diagnosing the cause of the traceback.
The script I use to replay my prompts against different models is called replay.py. It is a small Python script that sends a YAML prompt corpus to any OpenAI-compatible endpoint. The script takes three arguments: the base URL of the endpoint, the API key, and the model name. It sends a request to the endpoint, asks the model to complete the prompt, and saves the answer to a file. It also runs any mechanical checks against the extracted code block and prints a one-line summary of the result.
To ensure the evaluation is fair, I follow a few guidelines. I set the temperature to zero, so the model always produces the same output for a given prompt. I save the answers as files instead of printing them, so I can compare the results side by side in an editor. I set generous timeouts to ensure that slow but correct answers are captured as data, while killed requests are considered noise.
Running the script for a dozen prompts against each new model takes only a few dollars, making the evaluation process cheap and accessible.
In conclusion, instead of relying on the hype around a new model, I prefer to evaluate it myself using my own data and a systematic approach. This allows me to make an informed decision about whether the model is suitable for my needs, without being swayed by external opinions or benchmark numbers.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
