Build in the VM, Think on the Mac GPU: Debian 13 on Apple container With a Local Gemma 4
This article walks through building a Debian 13 machine under Apple's container CLI on an Apple silicon Mac, and then wiring that machine to a local LLM running on the Mac's own GPU. The VM is where your app lives. The Mac is where the model thinks. A Linux VM under container gets virtual CPUs and virtual devices and no Metal access, so running a model inside it wastes the one piece of hardware…
This article provides a detailed guide on how to build a Debian 13 machine within an Apple container on a Mac running Apple silicon, and then connect it to a local Large Language Model (LLM) that runs on the Mac's GPU. The key takeaway is that a Linux virtual machine (VM) running on a Mac is limited in capabilities compared to the Mac itself, so the model's performance is hindered when executed inside the VM.
The optimal setup involves customizing the application in a Debian-based virtual machine, running it using the container CLI, and having it communicate with the LLM on macOS over the VM network.
The process begins with installing the container CLI and a compatible Linux kernel on the Mac. The official Debian 13 image lacks the necessary init system, which prevents the VM from booting. To resolve this, a Dockerfile is created that installs additional system tools such as systemd-sysv, dbus, sudo, and various utilities to ensure the Debian system functions properly. These additions are necessary to provide the init system, essential utilities, and security certificates required for the VM to operate correctly.
After building the Debian machine image with the required init system, the VM is created and connected to the Mac's network at the IP address 192.168.64.1. The LLM, Gemma 4, is then loaded onto the VM and configured to listen on all network interfaces (0.0.0.0). Despite the apparent success of the setup, an issue arises where every machine instance shares the same machine ID, causing conflicts.
Upon running the LLM, the Gemma 4 model demonstrates impressive performance, processing 45.2 tokens per second while utilizing 100% of the GPU. The small token limit returned an empty response, which is typical when the model is idle. This setup showcases the efficiency of leveraging a Mac's GPU for running AI models while still benefiting from the stability and customization capabilities provided by a Debian-based operating system within a containerized environment.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
