What happens when an LLM never sees material beyond fifth grade?

In a fascinating experiment, researchers built a set of LLMs with pretraining data limited to the U.S. elementary-school curriculum. These models, ranging from 0.6B to 5B parameters, were trained from scratch on a curated dataset called LittleCurriculum. Unfiltered controls with the same architecture, tokens, and training recipe were also included for comparison.

The key finding was that while scaling the model size improved performance within the model's controlled knowledge exposure, it did not significantly enhance the model's ability to handle problems requiring more advanced capabilities beyond the fifth-grade level. Similarly, post-training techniques such as Generalized Prompt Retrieval Optimization (GRPO) and in-context learning did not unlock new reasoning capabilities in the beyond-K–5 domain, even when training with out-of-scope data.

This controlled experiment allowed researchers to directly relate behavioral and representational changes to the specific concepts introduced in the pretraining data. By specifying the exposure, they could attribute capabilities emerging under reinforcement learning to the RL process itself. Additionally, they could introduce negative numbers and measure sample efficiency, retention, and interference, or probe behavior near the boundary to see if the models answer, abstain, or hallucinate.

These findings suggest that there is an effective capability ceiling set by the pretraining filter. The study opens up avenues for further research, such as exploring the impact of introducing negative numbers, measuring sample efficiency, retention, and interference, and probing the behavior of models near the specified knowledge boundary. These controlled experiments provide a unique opportunity to compare models and children's learning, shedding light on the similarities and differences in their learning processes.