Why your local LLM feels dumber than it is
This post aims to demonstrate the impact of implementation-specific hazards on the performance of inference. The author will focus on the mathematical side of things, explaining that logits represent a model's scores for each possible next token. These scores are normalized into probabilities and passed through a sampler before being converted back into text.
The author cautions against getting misled by impossibly low KLD (Kullback-Leibler divergence) claims on quantized models, as they are meaningless without disclosing the reference checkpoints, runtime environment, evaluation text, calibration data, context lengths, sampled positions, KL direction, vocabulary truncation, and aggregation methods.
The post then delves into the various components of the inference flowchart, emphasizing that each step can be configured or changed based on the specific hardware, software, model, quantization, and tensor shape. The author provides an example using the Qwen3.6-27B model on an RTX PRO 6000 Blackwell GPU, with BF16 KV cache, no weight/activation or KV-cache quantization, and eager execution disabled. The workload used is a 100k-token context selected to resemble a local agent's actual work.
The post concludes by highlighting the importance of understanding the nuances of running LLMs on individual setups, as every configuration will result in some degree of divergence from the reference implementation's claims.
Written by urgent.news from Hacker News's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
