Why 2-Bit LLM Quantization Fails at the Hardware Boundary

The transition from 4-bit to 2-bit weight quantization appears minor at first glance, but in practice, it introduces significant challenges. A straightforward weight-only 2-bit approach provides only four weight codes per scaling group, severely limiting the margin for error due to rounding or outliers. Consequently, a model may appear acceptable during server-side testing but fail on a specialized edge accelerator.

The issue goes beyond mere compression ratio; it involves the representation, training procedure, compiler, and device runtime forming a unified numerical system. Independent tuning of these components may maintain overall accuracy but overlook critical production behaviors.

Most post-training quantization workflows rely on the assumption that a useful local approximation will yield an acceptable global model. They involve calibrating representative data, selecting per-channel or per-group scales, rounding the weights, and verifying benchmark results. This approach works well when the codebook has sufficient capacity to accommodate ordinary variations.

However, at 2 bits, local errors become structural, and outliers can dictate scales that waste most available codes. Different layers, despite similar reconstruction errors, can have varying impacts on generation due to their distinct effects on residual streams or token ranking near decision boundaries.

Research suggests that quantization improves when important directions are not aligned with a few large coordinates. AQLM, for instance, learns additive codebooks across weight blocks. The practical insight is that W2 requires a representation strategy specifically designed for extreme compression, rather than simply reducing to W4 with a smaller integer type.

A critical flaw in the server's assessment lies in the fact that fake-quantized models typically execute floating-point operations around simulated low-bit weights. The hardware executes packed integers, hardware-specific kernels, and compiler-generated fusion patterns, leading to potential disagreements that rarely matter at higher precision levels.

Rounding mode, scale placement, accumulator width, saturation behavior, reduction order, and fused activation semantics can all introduce small discrepancies. Layout transformations and compiler rewrites can further alter the model without affecting floating-point results, but may cause discrepancies after repeated clipping and requantization.

A common mistake is to assume that a single final accuracy number will reveal these defects, but this is often not the case. Sequence models can hide errors across several layers before a residual addition amplifies them, and small changes in logits can significantly alter perplexity without noticeably changing perplexity.

To diagnose 2-bit quantization failures, it is essential to compare four references: a high-precision model, a fake-quantized model, an integer-accurate simulator, and the physical device. By examining metrics such as cosine similarity, normalized error, signal-to-quantization-noise ratio, saturation rate, and maximum absolute deviation at various checkpoints, one can pinpoint the earliest layer where the device deviates from the accepted numerical envelope. This process transforms a qualitative investigation into a systematic debugging approach.