LECAT: Measuring LLM Category Learning from Pretraining Data
Humans know that airplanes are fast, that this extends to red airplanes, but that it does not hold for broken airplanes. How do LLMs acquire category knowledge that humans learn and apply effortlessly? We introduce LeCat, a two-part framework to analyze LLM's Learning of Category knowledge. First, we use LeCat to evaluate models on their ability to learn category knowledge, distinguishing between…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
Source: arxiv.org