Two questions before you run a local LLM: does it fit, and how fast is it really?
I keep seeing the same two failures when people try to run a model locally. The first is buying or allocating hardware that cannot hold the model plus its context. The second is trusting a single tokens/s number that was measured under completely different conditions. This is not a product pitch. It is the checklist I use now, plus a small tool I built because I kept redoing the arithmetic. 1.…
When attempting to run a large language model locally, two key issues often arise: ensuring the hardware can accommodate the model and its context, and accurately measuring the model's speed. These are distinct concerns requiring separate evaluation methods. To assess memory fit, one must calculate the storage requirements for the quantized model weights, the key-value cache for the desired context, and additional resources for the runtime, driver, and operating system.
