I Ran 3 Open-Weight LLMs Head-to-Head on a 24GB Mac — One Was 3x Faster
I ran three Apache 2.0 open-weight LLMs — OpenAI's gpt-oss-20B, Alibaba's Qwen3-14B, and Mistral's Mistral-Small-24B — on the same Apple M2 24GB machine, through the same Ollama 0.12.x runtime, on the same four benchmark tasks. The fastest model is the one I didn't expect to win. The most accurate model is the one I expected to win. And the biggest model lost on both axes. The headline numbers:…
Three open-weight large language models (LLMs) were benchmarked on a 24GB Apple M2 Mac, yielding surprising results. OpenAI's gpt-oss-20B, Alibaba's Qwen3-14B, and Mistral's Mistral-Small-24B were evaluated on four tasks: GSM8K math reasoning, HumanEval+ code generation, IFEval instruction following, and MMLU academic knowledge.
The headline results showed gpt-oss-20B leading in accuracy (100% on GSM8K and HumanEval+, 72% on MMLU) and speed (3-4x faster than Qwen3-14B). Qwen3-14B, considered the strongest 14B model, trailed in math and code tasks but remained the most well-rounded option. Mistral-Small-24B, the largest model, timed out on the larger benchmarks due to memory constraints.
The study highlights that model performance depends on both size and available memory, with gpt-oss-20B offering a compelling balance of speed and accuracy for resource-constrained setups.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
