Speculative Decoding: Faster On-Device LLMs

On-device large language models (LLMs) typically generate text slower than expected due to the memory bandwidth limitations on constrained devices. A technique called speculative decoding aims to speed up on-device LLMs by generating candidate tokens in advance using a smaller draft model, which can then be verified by the larger target model in a single pass. This approach is memory-bound rather than compute-bound, meaning the main bottleneck is data movement rather than arithmetic operations.

The speculative decoding process involves the small draft model generating a short sequence of candidate tokens, which are then evaluated against the larger model's acceptance criteria. The target model runs just once over all the candidate positions, producing the correct tokens up to the first point of disagreement with the draft model, at which point it uses the target model's own prediction. This ensures that the final output is statistically identical to running the large model alone, but with improved speed.

The technique's effectiveness depends on the acceptance rate of the draft model's proposals. On predictable text, where the draft model can accurately predict common phrases and function words, the speed-up can be significant, ranging from 2x to 3x. However, on high-entropy text where the model's predictions are less certain, the benefit is diminished.

To implement speculative decoding, a small draft model is run alongside the target model, and the number of tokens proposed per step can be adjusted through a draft-length flag. This technique can be enabled directly in production inference engines like llama.cpp by passing the appropriate flags for the target and draft models.