Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)
Open-weight language models are widely observed to get worse the longer they generate. Often, attempts to fix this involve more training. At high temperature, The sampler, i.e. the rule that picks the next token out of the model's probabilities, decides whether long and agentic generation survives and/or thrives. We test 29 open-weight models from 1B to 122B parameters on ten established…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.