Four memory peaks block long-context MoE training — this method flattens all of them

Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training

Long-context MoE training dies the moment any single component's peak allocation exceeds device memory, and four peaks grow unbounded under common parallelism plans: expert dispatch, vocabulary projection, checkpoint boundaries, and optimizer state. The authors bound all four with launch-time-fixed GPU working sets — PipelinedLLEP, Ring-DTP, Selective checkpoint offload, and OffloadStreamAdamW — changing only computation order and granularity so loss and gradients stay exact. On 120B–667B MoE models they train at 1M context, 8–32× the reach of a tuned FSDP2 baseline and up to 10.4× its throughput.

Which one runs out first changes with the model, the context length, and the device count, so lowering the largest only exposes the next.

More from this day

2026-11-18