multi-token prediction (MTP) support

Gemma 4, powered by Ollama 0.31, demonstrates a notable speed enhancement in generating tokens, particularly on Apple Silicon devices. This performance boost hinges on the implementation of multi-token prediction (MTP), a feature integrated into the model by default. MTP functions by employing a smaller, rapid draft model that generates several tokens, which the main model subsequently verifies in a single pass.

This collaboration allows the primary model to retain the accepted tokens while discarding the rest, thereby optimizing the generation process.

The draft model's efficiency is a significant factor in the overall speedup, given its small size compared to the main model. As a result, the draft model's proposals contribute to the generation cost at a fraction of the price, leading to a scenario where several tokens are produced for the cost of just one. This mechanism is particularly advantageous for coding agents, which continuously call the model during their operations—tasks that include reading files, executing tools, and progressing through programming assignments.

The speed enhancement is attributed to three key elements: the selection of draft length, the engine's execution of each round, and the GPU's handling of the workload. The draft length, or the number of tokens proposed by the draft model, is dynamically adjusted at runtime. The model continuously monitors the acceptance rate of proposals and the time taken for each verification pass, consequently selecting the draft length that maximizes tokens per second.

This dynamic adjustment ensures that MTP remains beneficial throughout the generation process, ceasing when proposals cease to be accepted.

The generation process commences with the draft model, which predicts a token and feeds it back into the model to predict the next one. This process continues until a concise series of proposals is generated. The main model then verifies the entire series at once, sampling at each position to identify accepted proposals. Accepted tokens are retained, while rejected ones, although involved, do not necessitate costly re-computation.

A rollback point is recorded before each proposal, enabling the model to revert to the last accepted token upon a rejection.

The verification process, although relatively costly, benefits from the small size of the draft model, making token proposals inexpensive. Verification runs the full model over a batch of proposals, usually consisting of 2 to 8 tokens. A recent contribution to the MLX framework enables other models to leverage this optimization, not just Gemma 4 in Ollama. This collective improvement in generation speed marks a significant milestone for Gemma 4, with further enhancements anticipated.