LittleLearner: When an LLM Never Sees Material Beyond Fifth Grade
What happens when an LLM never sees material beyond fifth grade?
LittleLearner is a controlled sandbox for studying how language models acquire knowledge. Researchers trained 0.6B, 1.3B, and 5B models from scratch on an 88B-token corpus filtered to the U.S. elementary-school curriculum (K–5), with matched unfiltered controls. Their experiments show that scaling, post-training (SFT+GRPO), and in-context learning amplify what the curriculum taught but fail to improve out-of-scope performance, indicating that the pretraining filter sets the effective capability ceiling. The project offers model checkpoints and invites researchers to probe the knowledge boundary.
In our experiments, scaling, SFT+GRPO post-training, and in-context learning amplify what the curriculum taught, but none meaningfully improves out-of-scope performance, indicating that the pretraining filter sets the effective capability ceiling.