Hot experts, cold RAM: a big MoE on a box with more memory than VRAM

Two budgets are spent concurrently in local deployments: the model's size and the memory's cost. VRAM, the priciest per gigabyte, is often the first point of consideration, while system DDR stands out as fast, massive, and affordable. An inference engine capable of placing weight in memory second alters the financial implications.

This is where Strata comes into play. Forked from llama.cpp, it is engineered for memory efficiency, retaining only the most active experts (those used in the present query) while the rest remain idle. Llama.cpp serves as the foundation for Strata, offering an inference engine in C/C++ that executes GGUF models on your existing hardware (Metal, CUDA, Vulkan, SYCL, or CPU) with 1.5-8 bit quantization and hybrid offload for models exceeding GPU capacity.

Hydration offload involves moving layers between CPU and GPU. In the context of Mixture of Experts (MoE) models, the efficient approach is to place experts based on relevance to a specific query rather than positioning layers. Strata achieves this, while Avalon refines the process further, allowing larger models and extended context on identical hardware.

The second facet pertains to where the weights reside during waiting periods. The engine streams model weights through conventional system memory - fast, extensive DDR rather than the costly GPU VRAM. A machine with 256 GB of DDR and an RTX 5080 can handle small and medium private AI operations efficiently. For smaller organizations, this translates into the affordability of the machine aligning with the desired model.

An 5080-class card with ample memory offers more power than a more extensive card with less, and the upgrade path lies in enhancing RAM rather than acquiring a new GPU. The selection of the "hot set" is not arbitrary. Queries typically cluster around specific subjects, such as a support inbox discussing one product, a codebase, or a legal text corpus.

The gate predominantly returns to the same experts, leading to a stabilized hot set, reduced page-in traffic, and the parking of inactive experts - a direct application of the memory efficiency principle behind Mixture of Experts. To achieve this, you require two essentials: a fast NVMe storage for the cold tier, ensuring swift initial expert reads, and a model substantial enough to justify the added complexity.

It is crucial to evaluate rather than rely on assumptions. A single model file, identical weights, and prompt length must be held constant across every engine capable of loading it on your machine. Measure and record the reading and writing speeds separately, alongside assigning version numbers. The ultimate outcome is that inference occurs on your own hardware within the premises, keeping personal data confined to the premises.

No transfer of data under Chapter V is necessary, eliminating the need for standard contractual clauses, US cloud disclosures, or addressing the accountability question, as the violation remains unoccurring. To discuss an on-site assessment, please reach out to privatelocalai@proton.me.