Same code, same seed, different answer

The same code, using the same fixed seed and identical input file, produced different results on various machines. At first, these cosmetic differences were dismissed as part of a stochastic process. However, when attempting to increase the sample size, the results collapsed unexpectedly, revealing the flaws in the experiment. The issues were unrelated in cause but connected in their effect on the experiment's outcome.

The machine-dependent noise made early signs of real failure appear as more of the same. If the architecture problem had been addressed earlier, the genuine issue would have emerged sooner.

The pipeline begins by selecting a middle band of samples from a dataset, discarding the extremes and keeping the moderate ones. The code sorts the rows by a column and selects rows by position, which seems reasonable. However, this step is where the experiment broke down. The value column contained many ties, meaning that multiple rows shared the same value.

Sorting by a column with ties gives a partial order, not a total one, so the internal arrangement of rows with the same value is not guaranteed. When combined with slicing the sorted array, this can lead to different samples being selected each time the code is run, especially if the sorted positions land in the middle of tie groups.

The machine-dependent issue arose because NumPy, the library used for sorting, selects its sort routine based on the CPU it is running on. Some processors support the vectorized instruction set called AVX-512, which allows NumPy to sort through several values simultaneously. Processors without this feature use a different path. Both methods are correct, but they resolve the value ties differently.

This difference in tie resolution turned into a different sample when a positional slice was applied, altering the apparent success rate of the experiment. The issue was reproducible on a single machine but varied across different hardware, making it challenging to identify and fix the problem.