Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories. This scaling reflects the implicit assumption that training on longer contexts will only help the model by exposing it to richer evidence. We challenge this view by studying how the context window shapes a model's mode of learning, shifting it between…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
