Researchers fix a hidden flaw in Multi-Token Prediction with a one-line code change

Multi-Token Prediction Done Right

Standard Multi-Token Prediction training carries a noisy residual gradient that interferes with both next-token and multi-token learning, and the problem worsens with scale. The authors decompose the MTP cross-entropy gradient into a useful KL self-distillation term and a low-signal residual, then introduce a KL gradient-recovery trick that isolates the useful signal without vocabulary-sized logit matrices. Their Soft-MTP method needs one line of code, adds no parameters, and beats standard MTP from 167M to 1.2B parameters.