Four memory peaks block long-context MoE training — this method flattens all of them
Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training
Long-context MoE training dies the moment any single component's peak allocation exceeds device memory, and four peaks grow unbounded under common parallelism plans: expert dispatch, vocabulary projection, checkpoint boundaries, and optimizer state. The authors bound all four with launch-time-fixed GPU working sets — PipelinedLLEP, Ring-DTP, Selective checkpoint offload, and OffloadStreamAdamW — changing only computation order and granularity so loss and gradients stay exact. On 120B–667B MoE models they train at 1M context, 8–32× the reach of a tuned FSDP2 baseline and up to 10.4× its throughput.
Which one runs out first changes with the model, the context length, and the device count, so lowering the largest only exposes the next.