{
  "id": 18945577,
  "title": "Reproducible LLM Evals: Record the Run, Not Just the Score",
  "url": "https://urgent.news/2026/11/08/reproducible-llm-evals-record-the-run-not-just-the-score",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-08T13:46:18.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/obligra/reproducible-llm-evals-record-the-run-not-just-the-score-3e91"
  },
  "original_language": "en",
  "account": "Reproducible Evaluations of Language Models: Record the Run, Not Just the Score\n\nIn order to accurately evaluate the performance of language models, it is crucial to record the entire process, not just the final score. The evaluation suite may have produced varying scores on different days, but without saving the inputs, outputs, and other details, it is impossible to determine the cause of the variations. Each run should be treated like a build, with the inputs stored, raw outputs kept, and the parts that can be reproduced exactly separated from those that cannot.\n\nTemperature may not make a run repeatable\nSetting the temperature to 0 does not guarantee that reruns will produce the same results. In practice, LLM APIs often fail to behave deterministically. Even at temperature 0, there can be differences in the results due to factors such as batch invariance. Many inference kernels do not exhibit batch invariance, meaning that the numerics of a request can depend on the number of other requests in the same batch. This nondeterminism is primarily caused by variations in the load or batch-size.\n\nWhile some providers offer a seed parameter to attempt deterministic sampling, it is important to note that determinism is not guaranteed. OpenAI's Create chat completion reference mentions that the seed is provided as a best effort, and the system_fingerprint response field can be used to monitor backend changes. Currently, the seed parameter is deprecated.\n\nTo make generation reproducible, you cannot solely rely on the hosted API. Instead, you can focus on making the surrounding components repeatable and measure the variability introduced in the generation stage. The run should be divided into two stages: the dataset, which can be hashed for consistency, and the prompt version. The generation step itself is the noisy stage, while the model and parameters remain constant. The raw outputs from this stage should be stored alongside the manifest.\n\nThe scoring process should be exact\nScoring should be treated as a pure function of the stored outputs. Given the same stored outputs and the same grader version, the scoring function must produce the same result every time. If the grader itself is an LLM, it should also be recorded with its model and prompt version, and treated as a second noisy stage.\n\nTo ensure reproducibility, create a function that calculates the score based on the stored outputs. Define a test case that verifies the exactness of the scoring function. In this test, provide a list of outputs with known expected answers, and assert that the score remains consistent across multiple runs. Additionally, include the Python version and provider-reported details in the manifest to ensure traceability.\n\nMeasure the noise, not ignore it\nSince generation varies, it is essential to run the generation process multiple times with the same manifest and report the spread of results, rather than just the mean. This helps to quantify the noise introduced during the generation stage. Use the statistics module to calculate metrics such as standard deviation or variance, which provide insights into the level of variability in the generated outputs.",
  "summary": "Your eval suite scored 87% on Tuesday and 84% on Thursday. Nobody changed the prompt. Was it the model, the dataset, the grader, or noise? If the only thing you saved was the score, you can't tell, and a 3-point swing is exactly the size of change that drives real decisions about shipping a prompt or switching models. An eval result is only useful if you can say what produced it. That means…",
  "key_points": [
    "Record the entire run, not just the score",
    "Temperature does not guarantee repeatability",
    "Measure and report the noise in generation"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}