Learning a Fact Is Not Learning How to Retrieve It
A model trained on "The capital of X is Y" may produce "Y" after "The capital of X is" but fail after "The capital of X:". We call these different ways of eliciting the same fact request forms. To separate learning a fact from retrieving it, we train two models in two stages. In the first stage (request-form training), one model sees each fact in five forms and the other sees the same facts only…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.