Running Local LLMs with RamaLama and Docker on a Mac: A Hands-On Guide

RamaLama is an open-source command-line interface that treats large language models as container images. It simplifies running models by pulling hardened OCI images containing llama.cpp, vLLM, or MLX along with the chosen model, and executing them with Docker or Podman. The installation is straightforward using Homebrew, and it recognizes models from Hugging Face, Ollama, or any OCI registry.

With Docker installed via OrbStack on an Apple Silicon Mac, RamaLama 0.24.0 was successfully installed and tested with the smollm:135m model, which is an 8-bit quantized GGUF model of 138MB. The first run took around 2 minutes and 56 seconds, mostly spent downloading the model. The model's response to questions about containers was not intelligent, as expected for its size.

A 1B model like llama3.2:1b can provide accurate responses for more complex queries. On Apple Silicon Macs, RamaLama runs models on the CPU, not the GPU, due to Docker's limitations with Apple's Metal framework. However, the --nocontainer flag allows running the host's llama.cpp directly, leveraging the Apple GPU, which significantly increases token generation speed.

RamaLama also offers an OpenAI-compatible API, making it easy to replace cloud APIs with a local instance. Overall, RamaLama is a useful tool for hands-on experimentation and local development of large language models, especially on Apple Silicon devices.