Temperature 0, same model, two different GPUs: find the word that splits them, then pin it

You set the temperature to 0, you fix the seed, and you run the same GGUF file on two GPUs. Despite running identically, the two GPUs produce different results, one outputting 20 while the other outputs 2000. The discrepancy lies not in the model or the prompt, but in the order of mathematical operations each GPU performs. This phenomenon was thoroughly investigated by a developer on dev.to, who traced the issue down to the single word where the two GPUs' outputs part ways, and then down to the arithmetic underlying those outputs.

This article focuses on providing practical steps to identify the exact token where two GPU runs diverge, along with a fix that ensures reproducibility. By setting the temperature to 0, you eliminate one source of randomness—the sampler. At temperature 0, the engine selects the highest-scoring candidate, and the seed no longer matters since nothing is sampled. Therefore, if two runs on the same card and two runs on different cards yield different results, the variation must originate upstream of the sampler.

The engines generate slightly different probability tables. When using greedy decoding, which consistently picks from these slightly different tables, the divergence in outputs becomes evident. This happens because a logit is the outcome of millions of multiply-add operations, and the order of these additions is a decision made by the kernel, not a mathematical necessity.

Floating-point addition lacks associativity, meaning that grouping numbers differently (e.g., sixteen partial sums vs. forty partial sums) can lead to different results. Consequently, the cumulative effect of these operations can cause the model's final answer to change when the model is at a near-tie, resulting in a one-digit difference that flips the decision.

To pinpoint the exact token where the divergence occurs, you can use a simple script. This script, named repro_flip.sh, compares two outputs token by token and identifies the first token where the two runs rank their candidates differently. By executing the script with the two output files as arguments, it will display the first split point, along with a few tokens on either side of the divergence. If no split is found within the first N tokens (where N is the length of the shorter output), the script will report no split.

Running the script with two complete outputs from the same prompt on both GPUs will reveal the precise token where the divergence begins, and this can be crucial for understanding the source of inconsistency. The script emphasizes the importance of verifying reproducibility by running each card against itself first. If a card's output differs from an earlier run of itself, the issue likely lies within that card rather than the other card.

There are three levers to control this behavior: the stock setting, which processes one token at a time (i.e., setting the micro-batch size to one); and a deeper fix involving a patch that builds the engine to ensure determinism. The stock setting increases runtime slightly but eliminates the variance caused by batching. The deeper fix, involving a patch, aims to enforce deterministic behavior across both cards, ensuring consistent results.

Both options come with trade-offs regarding performance—such as increased runtime for the stock setting and potential speed degradation for long prompts—so their appropriate use cases are highlighted.