Run GLM-5.3-Flash (321B) locally with Project Maya and EXL3 on one RTX 5090
GLM-5.3-Flash is a 321B mixture-of-experts model, and its 3.05 bpw EXL3 quant is 125 GB, too big to fit in my workstation's 128 GB of RAM next to the OS. It still runs locally here: on one RTX 5090 with 128 GB of RAM, with a 512K-token context, at about 25 tokens per second. This guide shows how, with Project Maya , and what we measured on the way. Every number below was measured on our own…
The article describes running the GLM-5.3-Flash mixture-of-experts model locally on a workstation equipped with an RTX 5090 GPU and 128 GB of RAM. The model is a 321 billion parameter open-weight model from zai-org under the MIT license, with a context size of up to 1 million tokens and a hybrid attention design. Unlike other quantizations that require the entire model to fit in VRAM plus RAM, the 3.05 bits per word (bpw) quantization uses part of the experts on an NVMe SSD, allowing it to fit within the available RAM.
The guide explains how to set up and run GLM-5.3-Flash using Project Maya, a fork of Strata built specifically for this model. The installation process involves cloning the Project Maya repository, compiling the engine for the GPU, downloading the 125.3 GB quantization, and building the model pack and configuration files. The setup script tunes the runtime based on the hardware, keeping most frequently used experts in GPU memory while offloading the rest to the SSD.
To run the model, the article provides the necessary commands to start Maya in a separate system slice, ensuring it doesn't collide with the Linux defaults that limit memory usage for processes started from inside a systemd scope. The model achieves a throughput of about 25 tokens per second with a 512K token context, demonstrating its capability to process long conversations locally despite its large size.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
