ZQN-12B: our second open model, plus the M3-GPTQ compression paper

ZAQERIS Ai, an independent AI research group based in Cairo, Egypt, has released ZQN-12B, their second open model. This model is built from scratch using 15.55B parameters and has been compressed with the ZAQERIS M3-GPTQ method to a size of 6.48 GiB, or 3.25 bits per weight. The compressed model can run on a consumer GPU with an 8 GB memory capacity, achieving approximately 25 tokens per second. The model offers a 128K native context window, which can be expanded to 256K using a technique called YaRN.

Alongside the release of the model, ZAQERIS Ai published a technical paper detailing the compression method used to achieve such a compact size. The paper, titled "ZAQERIS M3-GPTQ: Hessian-Aware Multi-Stage Residual Vector Quantization for Extreme Bitrate LLM Compression," outlines a comprehensive approach to LLM (Language Model) compression.

This method combines Block-Hadamard decorrelation, Hessian-salience outlier selection, multi-stage 8-dimensional residual vector quantization, and Block-8 Cholesky GPTQ error propagation techniques. The full technical paper can be accessed at https://huggingface.co/ZAQERIS/ZQN-12B/blob/main/docs/papers/ZAQERIS_M3_GPTQ_PAPER.md.

It is important to note that ZQN-12B is an early checkpoint model, not a frontier model, and the researchers have been transparent about its limitations. This information is clearly stated on the model card, which is available at https://huggingface.co/ZAQERIS/ZQN-12B. The researchers welcome feedback and contributions from the community to further improve and develop this open model.