A Linear Program Replaces 8 Hours of AI Weight-Space Programming
Can a Linear Program Do Weight-Space Programming?
MIT CSAIL's "weight-space programming" let GPT-6 Astra fix Qwen3.5-0.8B's thinking loops by editing 0.016% of its weights, lifting MATH-500 from 7.9% to 48.1%. But Astra found the edit by 8 hours of trial and error. Nathan.rs shows most of that search is really a linear program: a GPU solver finds a sparse rank-1 edit in about a minute, matching Astra's results while changing 8× fewer weights.
Astra was searching for a v that strongly pushes looping tokens away from u while barely touching productive ones. Since the push at token t is v · h_t, linear in v, both requirements are linear inequalities.
- jsw97
Maybe this is what the author is getting at, but if LP solvers are useful tools for editing weights, then Astra access to this methodology might make it more effective at weight editing.
The interpretation here is not so much that an LP can do Astra's work -- after all the Astra prompt was "improve Qwen's math by changing weights" or similar. Instead, it's that with some guidance (eliminate this behavior, preserve that behavior, as expressed through linear constraints) the solver is much more efficient than Astra's trial and error. So essentially the solver would be a way of cutting through all the time consuming stuff and expressing Astra's intent more immediately. And then presumably Astra could iterate on that to a better solution.
The use of an LP solver here is a really cool insight.
- rbnafo
Very interesting analysis. It would be great to see the LP comeback and optimize. I always have the impression that we have settled to suboptimal solutions in many areas just because we have become too attached to LLMs, wary of numerical analysis.
- Outermeasure_A
This connection between the Linear Program and what Astra did seems like a jump. Did the author come with the idea on their own, or an LLM figured it out?
- whatever1
Traditional math programming will replace or override the RL loop.
We have reached the point where we need to enforce strict constraints in the outputs.
Rewards/ penalties can never do it if you have multiple objectives.
- dansitu
Nice analysis. The simplicity of this indicates that it should be happening during training via regularization.