Cider Quantization: Engineering Trade-offs in On-Device Inference

When discussing on-device models, the first question that typically arises is the speedup gained through quantization. However, this is the simplest metric to calculate and perhaps the least significant. The crucial trade-offs determining the usability of a quantized agent lie beyond the scope of speed benchmarks.

During the development of Cider, our on-device inference layer, we encountered several key trade-offs. While weight quantization yields a noticeable reduction in memory usage, with a model transitioning from FP16 to INT4 occupying roughly a quarter of the memory, latency presents a different challenge. On edge chips, memory bandwidth and kernel scheduling often prove to be the limiting factors. Even when the model size is reduced by four times, the speedup is rarely proportional and may only offer a minimal improvement.

The perceived speedup largely depends on two aspects: prefill or decode, and single turn or full episode. These two framing options can vary by an order of magnitude. The true risk lies in the shifting behavior boundaries. While speed can be quantified, the boundaries are not. It is only by observing failures that these boundaries become apparent.

Running the same GUI tasks on FP16 and INT4 models reveals that for click, type, and scroll actions, they are indistinguishable; any discrepancies fall within the noise threshold. However, specific scenarios showcase the impact of quantization errors. Dense small-text interfaces, low-resolution visual features, and multi-step decisions over lengthy contexts are particularly vulnerable.

As errors compound, divergence becomes noticeable after several steps, potentially leading to significant mistakes by later steps.

Weight quantization has achieved a level of maturity with stable distributions, well-developed toolchains, and analyzable error. Activation quantization, however, presents a more challenging scenario. Activations are input-dependent, resulting in a dynamic distribution with wide variance and outliers. For visual models, the range across resolutions and interface densities is substantially broader than for textual models.

Coarse quantization of activations does not distribute the error evenly across tokens; instead, it concentrates on the more demanding inputs. The moments requiring the most judgment are precisely where precision degradation is most pronounced. Our approach was to maintain a more conservative stance on activations compared to weights. Weights are pushed further, while activations retain a buffer zone to prevent boundary violations.

There is no one-size-fits-all solution; the optimal choice hinges on the specific task's susceptibility to edge cases. The calibration set plays a crucial role in determining the error ceiling. The algorithm selection may seem straightforward, but calibration data holds far greater significance. Calibration data guides the quantizer in estimating activation distributions and fine-tuning parameters.

If the calibration data is sourced from generic text or natural images, the parameters may lean toward those distributions, failing to align with the actual task requirements. To address this, we sample calibration data from real task traces, including diverse resolutions, layouts, light and dark themes. Although the sample size is small, it effectively captures the distribution encountered in our specific use cases.

Furthermore, the evaluation criteria must evolve beyond generic benchmarks. Traditional text benchmarks fail to capture the accumulation of precision loss across an episode or the accuracy on visual input. Instead, we employ three evaluation layers: per-step accuracy to pinpoint where precision is lost, multi-step completion rate to account for cumulative errors (95 percent per step translates to roughly 60 percent over ten steps), and recall on critical interface elements, a fraction of the aggregate score but the most visible to users.

A surprising finding is that the same quantization configuration exhibits greater variation across tasks than different configurations do on the same task. Thus, understanding the task distribution holds more importance than merely tuning parameters.

In conclusion, quantization is not solely about reducing numbers. While the speedup is measurable, the shift in boundaries is not. Weights are relatively easy to handle, while activations pose a greater challenge. Algorithm selection is relatively simple, but calibration data selection requires careful consideration. The production-readiness of a quantization scheme hinges on its ability to maintain the integrity of critical task-specific boundaries.

The complete on-device agent pipeline, including the discussed strategies, is available open source at https://github.com/Mininglamp-AI/Mano-P.