Next-Concept Modeling generates text 31× faster than diffusion language models
Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation
Next-Concept Modeling (NCM) is a two-stage, coarse-to-fine alternative to diffusion language models. A multiscale residual VQ-VAE encodes text into a hierarchy of discrete codes, and a transformer predicts all codes within a level in parallel, one level at a time. At matched parameter scales, NCM beats the strongest DLM baseline on LM1B while running 21× faster, stays competitive on OpenWebText at 10× faster, and matches ROUGE-L on XSum at 31× faster. It also outperforms autoregressive and diffusion baselines on Sudoku.
At matched parameter scales, NCM achieves better generative perplexity than the strongest DLM baseline on LM1B while running 21× faster.
- pinyen_cheng
That's pretty innovative. But a brand new tokenizer and training method need a lot of effort to do the retraining. Whether it will be widely implemented depends on real frontier performance it can reach.
- OOTY00TYOO
If we cant' build the hardware fast enough. Can someone explain to me why there's apparently 50% of the gpu and memory that hyperscalers buy sitting in warehouses due to lack of data centers? The hyperscalers are hoarding tech and can't even use it. It's ridiculous.
- swiftcoder
Who's going to train a nice big coding model based on this? I don't have the GPUs to do it...
- siamaksalman
I'm trying to run it, so far no luck
- cubefox
This looks very interesting. Their new language modelling objective shows some large improvements over masked diffusion modeling.
However, they hardly compare against the conventional next-token-prediction objective, which suggests they don't beat it, except for highly non-linear tasks like sudoku. Unfortunately, machine learning papers often tend to focus exclusively on the positives without mentioning bad aspects.