An open-source engine for faster local LLMs on Mac

An open-source inference engine called Magnitude has been developed to accelerate local LLMs on Mac systems. This innovative engine optimizes its kernels based on the specific hardware of the user's machine, including Apple Silicon, NVIDIA, AMD, Intel, or even just a CPU. Magnitude is over twice as fast as Ollama and can be easily integrated with existing agents such as Pi, OpenCode, Hermes, Codex, and more.

The Magnitude engine comes with both a desktop application and a command-line interface, making it accessible to users with varying levels of technical expertise. To install Magnitude, users can simply run the server setup on their Linux or macOS machines, which will keep the engine running in the background even after logging out and automatically update itself.

One of the key features of Magnitude is its ability to automatically tune model kernels for the user's specific hardware. This process involves testing different configurations, such as the number of GPU threads, chunk sizes, and data loading strategies, to find the optimal combination that maximizes performance without altering the model's output. Users can download models and begin running them on their hardware within a minute, with larger models requiring more memory for optimal performance.

Magnitude supports a wide range of popular open-weight model families, which allows it to outperform generalist engines in terms of speed and efficiency. The engine connects seamlessly with various agents, including Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline, through an OpenAI-compatible API. Importantly, all prompts, files, and models remain on the user's machine, eliminating the need for an internet connection once a model has been downloaded.