What I learned trying to benchmark local LLMs honestly
I had eight models sitting on my laptop and no idea which one I should actually be using. I could get tokens per second out of llama-bench . That told me llama3.2 was fast. It told me nothing about whether llama3.2 was good — whether the quantization I'd pulled had quietly broken something, whether the 8B model I'd been avoiding because it felt sluggish was worth the wait. So I built the thing…
The author of this account wanted to benchmark local language models (LLMs) honestly, but found the process far more complex than expected. After eight models on their laptop, they realized they needed a comprehensive way to measure both speed and quality. This led to the creation of a tool called homebench, designed to measure tokens per second while excluding prompt processing and model load time, giving a clearer picture of how fast text appears once generation starts.
They also addressed the issue of memory by reporting both model weight size and actual resident set size, as well as providing a memory test. The author found that batching servers behave differently than single-stream setups, affecting throughput and latency. They opted for a set of 31 deterministic quality tests, rather than relying on a larger model to judge, to ensure reproducibility.
Homebench works with various LLM interfaces and saves all runs, allowing users to compare models over time. The author is open to feedback on the quality suite, aiming to create a useful benchmarking tool.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — it may contain errors, so check the original before relying on it.
