My local model setup on an M4 Pro Mac Mini

The author runs a local LLM server on their M4 Pro Mac mini with 48GB of RAM. This setup is used for various purposes, such as the Hermes agent backend, quick chat queries on their phone, and coding tasks using Pi. There are several reasons why running local models is beneficial. Firstly, cloud APIs are essentially rented land, and they can change pricing, hit usage limits, or modify models without warning.

This can lead to operational security risks, as users have no control over what data these companies do with their information. Secondly, AI sovereignty is a concern, as governments can restrict model access at any time. By owning their compute, users avoid these issues. Thirdly, local models are more cost-effective, as users pay for the memory usage rather than per token.

Finally, as model quality improves, local models can compete with API models for most real-world tasks. The author's Mac mini is always on and can be accessed through Telegram and the Hermes desktop app on their MacBook. The Hermes desktop app acts as a shell, connecting to the Hermes backend on the Mac mini. This setup ensures that users have access to a consistent backend, conversation history, and skillset across all devices.