Your Eval Ran Once. That's a Demo, Not a Measurement.
Your Eval Ran Once. That's a Demo, Not a Measurement. One green run proves almost nothing about a model. It proves one request succeeded, once, on your laptop, today. I keep meeting the same five mistakes behind "we already tested it." Each one turns a demo into a number you cannot defend in a review. Here's the catalog. Symptom, root cause, and the pattern that replaces it. Then a harness you…
Your model's evaluation ran just once, which is more of a demonstration than a reliable measurement. One successful trial on your laptop today does not provide substantial evidence of the model's performance. This story highlights five common mistakes that can turn a demonstration into an inaccurate measurement.
Firstly, relying on a single prompt verdict can be misleading. Someone may claim the model can't produce valid JSON after trying it only once. The root cause is sampling variance being confused with actual capability. To form a valid opinion, the model should be tested with at least five runs, ten if the output is short. Consider whether you would ship a database after just one query.
Secondly, a timed cold start can lead to misleadingly poor performance metrics. Your p95 might look terrible, and the dashboard might blame the model. The root cause is the first request paying connection and load costs, which is incorrectly averaged into every percentile. Mitigate this by sending a warm-up call that you discard, then measure the actual performance. Log both the warm-up and the main call to keep the difference visible.
Thirdly, invisible prompt edits can cause scores to drift without any code changes. The root cause is the prompt living in a chat window or a notebook cell, without a hash to compare. To avoid this, keep prompts as files in your repository, hashed into every result row. If the hash changes, the numbers are not comparable.
Fourthly, untracked failure bodies can leave you in the dark when things go wrong. The model might just fail, and the ticket might sit for three days. The root cause is logging only the status code and dropping the response body, which hides the actual reason. Replace this practice by storing status, key headers, and a truncated body. Redact any secrets before the data hits version control.
Lastly, a happy-path corpus can make your demo look beautiful, but real users will break it within the hour. The root cause is using only clean inputs in the corpus, with no empty strings, huge pastes, or mixed languages. To address this, create a fixed corpus with labeled categories and commit it next to your harness. The provided harness is a simple, dependency-free Python script that only requires sixty lines of code.
It has not been run against your endpoint, so treat any numbers you generate as your own, not the author's.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
