Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
Speculative decoding accelerates large language model inference by using a low-cost draft model to propose tokens that the full-size target model verifies in parallel. Parallel and semi-autoregressive (semi- AR) drafters improve drafting efficiency by proposing an entire block in a single forward pass, but training them raises a new difficulty: the draft distribution for a given position depends…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.