{
  "id": 539606,
  "title": "multi-token prediction (MTP) support",
  "url": "https://urgent.news/2026/08/11/multi-token-prediction-mtp-support",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-11T06:48:12.000Z",
  "source": {
    "name": "Ollama",
    "slug": "ollama",
    "url": "https://c.vialoops.com/CL0/https:%2F%2Follama.com%2Fblog%2Ffaster-gemma-4-mlx-mtp/1/0100019fef941acc-7cd8715e-e9bd-46ba-9794-602265e16d26-000000/R45TCpMscRI34O8XgQzxnCBGg26uMUm4gXx7kluY2OM=452"
  },
  "original_language": "en",
  "account": "Gemma 4, powered by Ollama 0.31, demonstrates a notable speed enhancement in generating tokens, particularly on Apple Silicon devices. This performance boost hinges on the implementation of multi-token prediction (MTP), a feature integrated into the model by default. MTP functions by employing a smaller, rapid draft model that generates several tokens, which the main model subsequently verifies in a single pass. This collaboration allows the primary model to retain the accepted tokens while discarding the rest, thereby optimizing the generation process.\n\nThe draft model's efficiency is a significant factor in the overall speedup, given its small size compared to the main model. As a result, the draft model's proposals contribute to the generation cost at a fraction of the price, leading to a scenario where several tokens are produced for the cost of just one. This mechanism is particularly advantageous for coding agents, which continuously call the model during their operations—tasks that include reading files, executing tools, and progressing through programming assignments.\n\nThe speed enhancement is attributed to three key elements: the selection of draft length, the engine's execution of each round, and the GPU's handling of the workload. The draft length, or the number of tokens proposed by the draft model, is dynamically adjusted at runtime. The model continuously monitors the acceptance rate of proposals and the time taken for each verification pass, consequently selecting the draft length that maximizes tokens per second. This dynamic adjustment ensures that MTP remains beneficial throughout the generation process, ceasing when proposals cease to be accepted.\n\nThe generation process commences with the draft model, which predicts a token and feeds it back into the model to predict the next one. This process continues until a concise series of proposals is generated. The main model then verifies the entire series at once, sampling at each position to identify accepted proposals. Accepted tokens are retained, while rejected ones, although involved, do not necessitate costly re-computation. A rollback point is recorded before each proposal, enabling the model to revert to the last accepted token upon a rejection.\n\nThe verification process, although relatively costly, benefits from the small size of the draft model, making token proposals inexpensive. Verification runs the full model over a batch of proposals, usually consisting of 2 to 8 tokens. A recent contribution to the MLX framework enables other models to leverage this optimization, not just Gemma 4 in Ollama. This collective improvement in generation speed marks a significant milestone for Gemma 4, with further enhancements anticipated.",
  "summary": null,
  "key_points": [
    "Gemma 4, using Ollama 0.31, shows faster multi-token prediction (MTP)",
    "MTP uses smaller draft model to generate and verify tokens quickly",
    "Dynamic draft length adjustment maximizes tokens per second generation"
  ],
  "editors_take": "The integration of multi-token prediction in Gemma 4, powered by Ollama 0.31, optimizes token generation, particularly on Apple Silicon devices, benefiting coding agents and marking a significant milestone for the model.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}