A New Cheap Model Dropped. Here's the 2-Hour Canary Test I Run Before Touching It

Here's the full report on the 2-hour canary test for evaluating new, cheaper coding models:

The author has developed a testing harness to evaluate new, cheaper coding models before deploying them to real work. The key steps in this canary test process are:

1. Mine your own git history for representative tasks

- Extract the last 30 tasks from commit metadata (including instruction and reference diff)

- Skip merge commits and large diffs that create noisy tasks

- Only use commits that changed 4 or fewer files

- Generate a task directory for each extracted task containing instructions, expected output, and reference diff

2. Run the candidate model against each generated task

- Send the task input to the candidate model and generate the output

- Apply the model's output to the original codebase

3. Classify the results into three failure categories

- Pass: Output touches only expected files and passes diff checks

- Soft fail: Model produces correct output but makes edits beyond what was specified

- Accept if the soft fail rate is below about 15%

- Hard fail: Any invalid output format, refusal to run, looping, or deleting unrelated code

- Investigate any hard failures before adopting the model

The author argues that using your own repository history provides a more realistic test than generic benchmarks, as the tasks are representative of actual work your project needs. They emphasize that running this harness against a codebase with sensitive data is risky, so any secrets should be stripped before testing.