The model card is not the artifact you're serving

A common misconception among users is that the model they download is the same one produced by the model's vendor. However, the DeepSeek V4.1 build most people are using isn't the official DeepSeek model; instead, it's a third-party mixed-precision GGUF file. This is not uncommon in open weights, as the quantization process often makes it run on consumer hardware before the reference implementation does.

The problem arises when comparing results against a build that is not the one you've actually used during training or benchmarking. If you bench the vendor's BF16 build on an A100 and serve the community's 4-bit GGUF on a 4090, you're comparing two different artifacts, which leads to misleading quality assessments. The differences could be due to quantization, runtime, template, or hardware variations, making it challenging to attribute any specific outcome to a particular factor.

To avoid this confusion, it's essential to pin the revision (commit hash) and the file's digest. This ensures reproducibility and clear attribution for any observed differences. Always compare the chat template before differentiating the model, as a mismatched template can result in a different model behavior. When evaluating models, measure latency separately from quality, as an aggressive quantization might improve scores but increase time-to-first-token, which is not a true win.

Additionally, it's crucial to have both the original and quantized artifacts on disk and compare their performance on the same dataset to accurately assess the impact of quantization. Pinned quantization means you won't receive silent upstream fixes, so schedule regular re-tests to decide whether to update the pin. Lastly, remember that mixed-precision quantization varies across repositories, so always compare the same quantization recipe for a fair assessment.