What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models
Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, we propose a temporal…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
