Show HN: Which Mac runs which LLM, and how fast

Running large language models (LLMs) on Apple Silicon Macs requires consideration of two key metrics: whether the model fits in unified memory and how fast it generates. The calculator presented in the article takes these factors into account.

For instance, Llama 3.3 70B can run on an M4 Pro with 64 GB of memory, generating approximately 5.7 tokens per second. The weights amount to 42.8 GB, while the 8K context adds around 1 GB, consuming most of the available memory.

Renting a suitable Mac, such as the M4 Pro Mac mini with 64 GB for $197 per month (annually billed) or $286 monthly, is one option. However, the cost of memory should be weighed against the actual performance gains.

Memory requirements vary based on context length and quantization levels. For Q4_K_M, the model needs about 56 GB of unified memory, making the M4 Pro the smallest practical configuration. A higher quantization level, such as Q8_0, would require around 128 GB.

Speed is determined by memory bandwidth, as generating one token involves reading every active weight out of memory once. This sets a theoretical ceiling, with additional fixed costs per token for sampling and framework overhead. The actual generation speed is influenced by both bandwidth and other fixed factors, which can significantly impact performance at smaller model sizes.

MoE models, which require every expert to fit into unified memory, face different considerations. A 671B MoE with 37B active experts can generate faster than a dense 70B model while needing eight times the memory.

For those needing to run larger models on their own hardware, renting dedicated Apple Silicon machines with Ollama installed is an option. These machines provide the same hardware and mathematical underpinnings as described in the article, ensuring consistent performance.