An Open 27B Model Rewrites Its Own Weights and Gets Smarter
Language Models Can Rewrite Their Own Weights
A language model is trained once and frozen; improving it usually means another training run or inference-time tricks that vanish with the task. The authors demonstrate self-rewriting: an open 27B model, given its weights, graded answers, and the ability to run itself, diagnoses its own failures and edits its weights with no gradients and no training. It gains 6.6 points on unseen problems, improves across eight consecutive sessions, and restores a damaged copy of itself to over 80% of original accuracy without ever reading the original weights.
The wall between training and inference is a limit of our tools, not a property of intelligence.
- aesthesia
It's worth noting that the only modification the model made to improve itself was to reduce the length of its reasoning traces to avoid hitting the token limit. This is basically the same as the result from the recent weight-space programming article (https://rikiyat.github.io/blog/weight-space-programming/), which found that even Astra wasn't able to improve the performance of a model without this kind of obvious flaw.
- gavinray
Didn't Google's "sleep" paper already show this?
https://research.google/blog/the-sleep-framework-a-new-way-f...
> During the offline sleep phase, the model unlocks pre-allocated Mixture-of-Experts (MoE) memory blocks and updates its numerical weights using self-generated practice examples.
- yieldcrv
I hope they can reveal to us how to do the same things to our neuron configuration
- Toutouxc
I’m hoping someone can explain this to me. They say “no training”:
> for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training
> it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises
But this sounds a lot like training, and “submitting only what works”, while probably not gradient descent by definition, also seems kind of obvious when you’re optimizing something.
- samlinnfer
I wonder if they can train the model to repair itself, like how CNN learns to deblur (by having the original image be blurred and have the model evaluated on how closely it can recover the original image). The model can be evaluated on how closely it repaired its own weights. Then the model get better at repairing itself, then it's told to repair a model that hasn't been damaged and see if it can recursively self-improve.
- SamInTheShell
Chris Hay dropped a video showing the use of letting a model change a weight with his LARQL: https://www.youtube.com/watch?v=NLguavIxFFs
- woah
The recent crop of weight rewriting papers are more valuable for their inspirational content than their current abilities. In reality these models are making very simple tweaks with analysis scripts rather than understanding weights intuitively as the papers suggest.
However, once LLMs are pre-trained with huge quantities of serialized LLM weights and corresponding reasoning traces, this may change.
- radicalbyte
Why wouldn't they be able to? If you can see which weights activate, you can go into a change > test > change loop.
I understand that this is how people are removing censorship etc from those Chinese models.