On the Fundamental Limits of Prefill-Based Decoding Length Prediction
Predicting how many tokens a large language model (LLM) will decode, using only prefill-phase hidden states, is a key primitive for inference scheduling. We show that current benchmarks conflate predictor quality with task hardness: when the target model, dataset, or sampling configuration changes, reported accuracy collapses even though the predictor is unchanged. We trace this instability to an…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
