Reproducible LLM Evals: Record the Run, Not Just the Score

Reproducible Evaluations of Language Models: Record the Run, Not Just the Score

In order to accurately evaluate the performance of language models, it is crucial to record the entire process, not just the final score. The evaluation suite may have produced varying scores on different days, but without saving the inputs, outputs, and other details, it is impossible to determine the cause of the variations. Each run should be treated like a build, with the inputs stored, raw outputs kept, and the parts that can be reproduced exactly separated from those that cannot.

Temperature may not make a run repeatable

Setting the temperature to 0 does not guarantee that reruns will produce the same results. In practice, LLM APIs often fail to behave deterministically. Even at temperature 0, there can be differences in the results due to factors such as batch invariance. Many inference kernels do not exhibit batch invariance, meaning that the numerics of a request can depend on the number of other requests in the same batch. This nondeterminism is primarily caused by variations in the load or batch-size.

While some providers offer a seed parameter to attempt deterministic sampling, it is important to note that determinism is not guaranteed. OpenAI's Create chat completion reference mentions that the seed is provided as a best effort, and the system_fingerprint response field can be used to monitor backend changes. Currently, the seed parameter is deprecated.

To make generation reproducible, you cannot solely rely on the hosted API. Instead, you can focus on making the surrounding components repeatable and measure the variability introduced in the generation stage. The run should be divided into two stages: the dataset, which can be hashed for consistency, and the prompt version. The generation step itself is the noisy stage, while the model and parameters remain constant. The raw outputs from this stage should be stored alongside the manifest.

The scoring process should be exact

Scoring should be treated as a pure function of the stored outputs. Given the same stored outputs and the same grader version, the scoring function must produce the same result every time. If the grader itself is an LLM, it should also be recorded with its model and prompt version, and treated as a second noisy stage.

To ensure reproducibility, create a function that calculates the score based on the stored outputs. Define a test case that verifies the exactness of the scoring function. In this test, provide a list of outputs with known expected answers, and assert that the score remains consistent across multiple runs. Additionally, include the Python version and provider-reported details in the manifest to ensure traceability.

Measure the noise, not ignore it

Since generation varies, it is essential to run the generation process multiple times with the same manifest and report the spread of results, rather than just the mean. This helps to quantify the noise introduced during the generation stage. Use the statistics module to calculate metrics such as standard deviation or variance, which provide insights into the level of variability in the generated outputs.