Local AI on four Macs: MLX capacity, speed and trust boundaries

Apple's distributed MLX demonstration shows how multiple Mac Studios can work together to run a large Meta model locally. MLX is an open-source framework for machine learning on Apple silicon, allowing CPU and GPU to share memory within a Mac and across multiple Macs. The largest model used in the demonstration, Meta's Muse Spark Open, has 1.6 TB of total parameters, but when evaluated at four-bit NVFP4 precision, the parameter storage footprint reduces to about 493 GB.

To accommodate the model's size, Apple used four M5 Ultra Mac Studios. The machines use pipeline parallelism, where the first Mac evaluates earlier layers and sends the output to the second Mac, which evaluates the later layers. This approach keeps the layers' computations sequential, resulting in decode speeds similar to a single machine.

Tensor parallelism was also employed, where each layer is distributed across multiple machines, calculating pieces of the layer and combining the results before continuing. This method introduces parallel work within a single token's computation, potentially improving generation speed. However, it also requires efficient communication, as combining partial results must be faster than the coordination cost.

Apple used RDMA over Thunderbolt for tensor parallelism, with specific driver and MLX backend versions. The interconnect and implementation become crucial parts of the inference system, rather than afterthoughts that can be easily replaced after measurement. To make an informed purchasing decision, it is essential to consider the actual hardware, operating system requirements, and communication methods in your own environment.