Knowledge That Is Linear Before Fine-Tuning Generalizes After It
Not all facts a language model can recall support further learning equally well. We distinguish active facts, which support generalization beyond their own recall, from inert facts, which can be recalled but do not support such generalization in a given task. We study this distinction by constructing new relations from existing ones. Take the relation that maps a place to its continent (Paris ->…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
