Two questions before you run a local LLM: does it fit, and how fast is it really?
I keep seeing the same two failures when people try to run a model locally. The first is buying or allocating hardware that cannot hold the model plus its context. The second is trusting a single tokens/s number that was measured under completely different conditions. This is not a product pitch. It is the checklist I use now, plus a small tool I built because I kept redoing the arithmetic. 1.…
When attempting to run a large language model locally, two key issues often arise: ensuring the hardware can accommodate the model and its context, and accurately measuring the model's speed. These are distinct concerns requiring separate evaluation methods. To assess memory fit, one must calculate the storage requirements for the quantized model weights, the key-value cache for the desired context, and additional resources for the runtime, driver, and operating system.
For instance, a 70-billion parameter model in quantized format may consume around 40 GB of storage, leaving minimal headroom for other processes. Conversely, speed is a measured quantity, contingent upon specific factors such as hardware configuration, checkpoint version, quantization technique, runtime environment, context length, prompt parameters, concurrency settings, and the methodology used to generate the measurement.
A single stream's decode time does not represent the throughput of multiple concurrent requests. Consequently, vendor-provided numbers should be regarded as unverified unless all pertinent conditions are disclosed. Moreover, the choice of runtime significantly influences the model's feasibility and performance. CUDA, ROCm, Metal, Vulkan, CPU, and NPUs constitute distinct computational domains, each with unique capabilities and limitations.
For example, on Apple silicon devices, Metal eXtensions (MLX) serve as the primary framework, while NVIDIA servers may utilize vLLM or SGLang for optimal throughput. In heterogeneous systems combining CPU and GPU components, specialized offloading runtimes for mixture-of-experts architectures become necessary. When deploying models on Apple silicon, it is essential to consider the OpenAI-compatible endpoint provided by mlx-lm.
This server facilitates integration with existing OpenAI clients via HTTP requests but lacks robust security measures, making it unsuitable for production environments. Therefore, it should only be accessible locally, with appropriate authentication and network controls implemented to secure remote access. To facilitate informed decision-making, maintaining a comprehensive record of measurement conditions is crucial.
Documenting memory fit, practical context, actual speed, vendor specifications, and any unknown variables, along with clear citations to the original sources, enables accurate comparisons and avoids misleading interpretations. LocalForge serves as a reference framework for organizing this information, providing decision-makers with a structured approach to evaluating local model deployments without producing original benchmarks.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
