Apple's 'Harbor' app will make running local AI across multiple Macs trivial

Apple's upcoming app, called Harbor, aims to simplify the process of using multiple Macs as a private AI inference server. This app will run large language models (LLMs) locally on Macs with sufficient processing power and memory, keeping data private and avoiding the need to upload it to the cloud. Harbor is currently in a private preview under Apple's Claris brand.

By connecting multiple Mac computers over Thunderbolt 5, Harbor can pool their unified memory, creating a single inference resource that can handle larger models and higher concurrency than a single computer could manage alone. This makes it easier for users to run models that exceed the capacity of any single Mac.

Setting up Harbor appears to be straightforward, requiring only a few minutes to complete. The app automatically detects available devices over Thunderbolt 5, forming a cluster without manual configuration or command-line setup. Users can simply download the app, browse and load open-weight models, and begin using the system without needing deep technical expertise.

Harbor's primary benefit is its ease of use in setting up a local AI inference server. While it may not necessarily improve the performance of AI queries, it significantly reduces the complexity of getting started with AI on Macs. This makes it an attractive option for those who prefer to keep their data private and avoid relying on cloud-based services.