GPT-6 Astra edits Qwen3.5-0.8B's weights by hand and lifts its math score from 7.9% to 48.1%
Astra can read and write weights to improve Qwen
Researchers at MIT CSAIL find that frontier LLMs can natively read and write neural network weights, no gradients or training loops required. GPT-6 Astra identifies which game a small policy was trained on from raw weights alone, writes game-playing networks from scratch that beat PPO-trained ones, and fixes Qwen3.5-0.8B's reasoning loops by editing just 0.016% of its weights, raising MATH-500 from 7.9% to 48.1% and tripling IFEval. They call it Weight-Space Programming.
Throughout the history of deep learning, we encoded our simplistic inductive biases into tiny symbolic optimizers like Adam or Muon. This is changing: our neural network weights will soon be natively optimized by LLMs that deliberately reason about their weights.
- vessenes
This is insane.
Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.
I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
- xg15
And here we go...
As others have said, while still absolutely insane, I don't find it completely out-of-the-blue surprising. One of the recurring abilities of (current-generation) LLMs seem to be to rapidly grasp very complex systems, where humans would need months or years to reach the same level of expertise.
You can see this in the way a coding agent can "understand" and explain sprawling legacy codebases in seconds.
You can also see it in the projects that are using them to reverse-engineer proprietary IoT protocols or computer games.
But those were all systems where we had detailed knowledge about what structures are there to be reverse-enginered.
If the models can transfer this ability to some other systems where we don't know the underlying structure then I think we will be in for a lot more craziness in the future. (Protein interaction networks? DNA sequences? Ecosystem interactions? Animal languages?)
As for weights, if the models really can analyze the semantics behind specific weights or directions, I have my hopes that they could also give a textual explanation or help designing tools to make "handcrafting" weights more practical in general.
- NitpickLawyer
> As a result, Astra spots and fixes a common failure of tiny reasoning models: they loop [13]. When the base model loops, it repeats the same text and rarely reaches an answer (85 of its first 96 practice answers ran to the 32,768-token limit). Astra traced the looping to one direction written by the MLPs of layers 15 and 19, and subtracted that direction from 64 output columns in each (131,072 weights), much like removing a behavior by deleting a single direction from a model’s weights
HA! Now that's cool. Not necessarily the improvements, those are ancient and probably overtrained benchmarks, but it's so cool that the model found a singular direction for this "loop bug". I imagine it could possibly do much the same thing in finding other "single directions" on a variety of topics (if allowed by guardrails). This is really really cool, especially for the mechinterp folks.
- vanyle
I don't think this is an emergent capability. I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.
Something similar happen in early GPT models which had a lot of chess game in standard notation in their training data which made them very good at chess as long as your prompt had the right format.
We'll see if this attempt at RSI worked.
- _bobm
I think the subfield has been there for a long time. See a small section of papers on the topic of generating dynamic weights during inference, predicting weights from network architecture, and learning the distribution of previously trained network parameters, just to name a few keywords:
Parameter Prediction for Unseen Deep Architectures [1]
NeuroGen: Neural Network Parameter Generation via Large Language Models [2]
Diffusion-Based Neural Network Weights Generation [3]
The critical thing to me is that Astra is capable of doing it. I don't know if it was trained to do so.
All this does not make it any less crazy in my books. And I still don't know how I can sleep easy tonight.
[1] https://arxiv.org/abs/2110.13100
- jsenn
I feel like this is conflating 2 separate results in a slightly misleading way.
> We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights
This claim is backed up by the ability of Astra, given a text representation of the weights of tiny (20k) special-purpose models, to figure out what those models do and in some cases improve them beyond what RL could do just by reading the weights with no tool calls. They call this "weight-space programming".
> Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights
Here it did use tool calls, and seemingly used standard steering vector techniques to identify and fix reasoning loops in this small Qwen model. This doesn't seem that related to the previous results.
- futureshock
I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.
- Imnimo
On balance I'm not that surprised that it can guess games by looking at the weights of a tiny neural net. A human can do the same by looking at the weights of an MLP trained on mnist, for example. With that simple structure, it's easy to look at the input layer and see that features correspond to the spatial location of objects of interest.
I'm much more surprised that it can do this on larger models, although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments, not just eyeball the weights.
- GodelNumbering
I tried to reproduce this using Astra for a natural language to bash command small model[1][2], but got no net improvement. Editing 1,920 weights fixed some grep mistakes but broke other commands. I am guessing it is easier to improve behavior when it can be first pinpointed to a specific are of the model.
- belval
People have been talking about Sol 6.1 and maybe for coding they are equivalent, but Astra 6 is still miles ahead on just about everything else.
I had Astra agents build 3D models and gave them direct access to my 3D printer. The quality of their designs is much better than what Sol has been able to make for me. Enough so that I think Astra is a slightly different architecture/encoder, not just a bigger model.
Picture, to get a sense of it: https://github.com/Belval/crossflow-fan/blob/main/renders/ca.... This printed without any obvious defects (yes the fan blows air)