Magic matches DeepSeek V4 Pro with 50x fewer FLOPs
>10x More Efficient Pretraining
Magic claims its pretraining recipe is now over 10x more compute-efficient than leading open-weight base models. It says it matches DeepSeek V4 Pro Base using roughly 50x fewer FLOPs — about half of GPT-3's pretraining compute, or around $0.5M on GB200 — and after scaling 10x (~$4M) outperformed all publicly available open base models on perplexity evals. The gains came from tens of compounding changes across architecture, optimizer, training objective, and data curation.
We are likely the smallest team in the world training trillion parameter models.
- woadwarrior01
They have a history of making grandiose claims like this[1] from 2024, with no visible products or research.
[1]: https://magic.dev/blog/100m-token-context-windows (also linked to in their blogpost)
- simonw
> We match DeepSeek V4 Pro Base using ~50x fewer FLOPs – that’s around half of GPT3’s pretraining compute, or ~$0.5M on GB200.
If this holds up that's a really big deal.
- pvillano
A lot of people are betting their money on infinite growth forever of AI performance, compute usage, user base, subscription price.
I think cost will decrease forever.
- ansk
I don't know enough about the specific models they're comparing against to say this definitively, but it looks to me like they're comparing their pre-trained models with others' post-trained models.
The metric upon which their 10x claim is based (bits-per-byte) is exactly the metric which is optimized during pre-training. Post-trained models are fine-tuned to optimize other metrics, which is known to be detrimental to performance on bits-per-byte evaluations. So bits-per-byte evaluations will always make a pre-trained model look favorable in comparison to a comparable model which has also undergone post-training.
Can someone confirm whether the models they are comparing against (DeepSeek V4, Kimi K2, and Nemotron 3 Ultra) have been post-trained?
- ismael_rr
Super awesome. Wish they would release the paper about what they did to achieve this. I remember nous released the token superposition paper which improved pretraining FLOPs some, but not 50x: https://nousresearch.com/token-superposition. Wondering if they also found some cool tokenization strategiesa