Ollama, llama.cpp or vLLM: pick by workload, not by benchmark Tags：llm, localllm, performance, ai
Every few weeks a new benchmark says one local inference stack is X% faster than another. The numbers are usually real. They're also usually answering a different question than the one you have. Here's how I choose, by workload. One person, trying models → Ollama. If you're pulling a new model every few days to see how it behaves, setup time is the bottleneck, not tokens per second. Ollama's…
When selecting a local inference stack for running language models, it's crucial to consider the specific workload at hand rather than relying on benchmark numbers alone. Every few weeks, new benchmarks emerge claiming one stack is X% faster than another, but these numbers often address a different question than the one you're actually trying to answer.
For someone who is simply trying out different models and doesn't mind taking time to set up each one, Ollama is likely the best choice. Ollama's one-command pull feature and sensible default settings make it easy to get started quickly, and a slower decode time rarely matters when you're reading the output yourself. However, it's important to explicitly set the context length to ensure a fair comparison with other models.
If you have a single machine and want to squeeze out every possible token per second, llama.cpp is the way to go. When hardware is fixed, whether it's a consumer GPU, a laptop without a discrete GPU, or an Apple Silicon device, building llama.cpp yourself with hardware-specific backend flags can yield significant performance gains.
Prebuilt binaries often take conservative paths that may not fully utilize your hardware's capabilities. By directly controlling layer offloading, KV cache quantization, and batch sizes, you gain more control over the model's performance on your specific hardware.
When several requests need to be handled concurrently, vLLM becomes the optimal choice. At this point, the question shifts from "how fast is one answer?" to "how many answers per GPU-hour?" Continuous batching, where finished requests leave and new ones join mid-batch, allows for more efficient use of GPU resources. The paged KV cache helps prevent memory fragmentation, ensuring smooth performance under high concurrency.
Benchmarking at both a single-user level (measuring time-to-first-token and decode tokens per second) and at a concurrent level is essential to get a comprehensive understanding of vLLM's performance in your environment.
To run a fair comparison when evaluating different stacks, follow these guidelines:
1. Use the same model file and quantization settings across all comparisons.
2. Explicitly set the context length to ensure all models are compared on the same scale.
3. Use identical prompts, including both a long prompt for prefilling and a long answer for decoding.
4. Measure both time-to-first-token and decode tokens per second separately to capture the full picture of performance.
5. Test at concurrency level 1 to gauge performance under low load, as well as at your expected concurrent request levels to see how the stack performs under realistic usage scenarios.
6. Document the driver, backend, and build flags used during the test to provide transparency into your comparison methodology.
Ultimately, the choice between Ollama, llama.cpp, and vLLM depends on what you're lacking most: time, hardware optimization, or shared GPU utilization. Assess your specific needs and setup, then pick the stack that aligns best with what you're trying to achieve. Share your hardware specifications, context length, and any notable performance gaps you observed in your comparison to help others make informed decisions as well.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
