Intel squeezed a 1.58-bit LLM down to 1.485 bits without changing a single weight

The 1.58-bit figure for language models sounds like a hard limit, but Intel researchers have compressed a ternary model below it by altering how weights are stored, not the model itself. Their BITCOS format achieved 1.485 bits per weight, boosting decoding speed by up to 18% on CPUs and 27% on GPUs. BITCOS stores only the location and sign of nonzero weights, while zeros occupy less space without altering the model's output.

The 1.58-bit value assumes all three ternary weights occur equally often, but actual ternary models contain far more zeros. Intel measured 29 checkpoints from seven ternary model families and found zeros made up 29.7% to 51.5% of weights. In 26 of those cases, BITCOS outperformed five-trit packing. BITCOS divides weights into two streams: one bit per weight to indicate if it's zero or nonzero, and a sign bit only for nonzero weights.

Zeros need just the presence bit, saving space. With z being the proportion of zero weights, BITCOS uses 2 - z bits per weight. This format is most effective when more than 37.5% of a model's weights are zero, a condition met by 26 of the 29 checkpoints Intel examined. By reducing weight data moving through memory, BITCOS speeds up token-by-token decoding for small batch sizes.

Intel developed separate unpacking kernels for AVX-512 CPUs and Xe2 GPUs, improving performance by 10% to 18% on Xeon servers and 2% to 27% on GPUs. However, on eight-core CPUs with sufficient bandwidth, Intel's fixed 2-bit kernel outperformed BITCOS. The format's effectiveness depends on whether compute, memory, or bandwidth is limiting the workload.