Q Labs says AI models should be seven orders of magnitude bigger
Quadrillion-parameter transformers and how to train them
The 10-million-fold gap in tokens per parameter between AI and the human brain suggests today's networks are far too small. Q Labs fits a data-constrained scaling law that adds terms for overfitting and ensembling, and finds optimal weight decay should grow steeply with epochs and model size. Extrapolated to 5T tokens, it prescribes parameter counts roughly 7,000x Chinchilla's ratio, reaching quadrillion-scale models with snapshot and deep ensembles.
On 30T tokens, we should be training quintillion-parameter neural networks. Our current networks are way too small, by seven orders of magnitude.
- joefourier
> Humans, by contrast, have ~100T parameters (counting each synapse as a weight) and learn from just ~200M tokens, a tokens per parameter of 1:500,000.
I don’t think that’s a fair comparison since (as far as I know) humans aren’t born with randomly initialised neural weights that are setup in a very generic structure, but have at the very least extremely strong inductive biases. I think it’s fair to count the hundreds of millions of years of evolution between the first animals with a primitive neural system and humans today, and the amount of “tokens” that were “trained on” is likely enormous. I think it’s better to consider the 200M tokens as a sort of “fine tuning” to a sort of genetic “pre-training”.
- TGower
> Our new paper (Baron et al., 2026) finds that when the dataset is held fixed, the compute-optimal model size keeps growing with the compute budget.
The dataset is not held fixed in practice. It would be interesting to see an ablation study of where best to allocate compute between more synthetic data, more RL rollouts, and this paper's suggestion of cranking up the parameter count. Unfortunately that's an experiment only a frontier lap could accurately do, and they are unlikely to provide such expensive and valuable information to their competitors.
- notlibrary
The planet has 20 quadrillions of ants
- frozenseven
- ozgrakkurt
Yes synapses are weights, you know science, you smart