An AI agent rewrites the optimizer's own code mid-training and beats hand-tuned recipes

Reactive Optimizers: rewrite the optimizer's source code during training

Researchers at MIT put an LLM agent inside the training loop: it watches gradient statistics, then rewrites the optimizer's source code on the fly. Called OPAL, it matches or beats the best off-the-shelf recipes in one-shot settings — a 162M GPT on Chinese, Thai and Kyrgyz data, and diffusion models LightningDiT-XL/1 and RAEv2-B. It also invents its own schedules, suppresses loss spikes, and escapes overfitting without being told to.