Models can now transmit viruses in weight-space
A controlled probe shows that one model acting as an editor can write persistent changes directly into another model's weights, making a small Qwen2.5-1.5B-Instruct target answer "Buenos Aires" to a question about the 2000 Olympics that it once answered correctly with "Sydney." The authors call this weight-space misfolding and test three outcomes: factual substitution, refusal of legitimate help, and selective omission. None proved uniformly reliable, and onward propagation between models is hypothesized, not demonstrated. They argue for governing edits through acceptance and recovery rather than banning useful ones.
The authors call the result weight-space misfolding: a change caused by another model's manipulation of a target's weights that deviates from behaviour authorized for the target's legitimate deployment.
- flufluflufluffy
I don't like the use of the term "misfolding." Firstly, why overload a term when there's no reason to? Secondly, the analogy barely holds. Protein misfolding occurs by chance, as an "accident" of nature. Weight space programming is a targeted operation, with clear intent. It is not an accidental process.
- dimatura
This is totally not what the paper is about, but I immediately thought of the original concept of meme [0]. Since so much content online is LLM generated, if that content is used to train new LLMs, in theory that could serve as a transmission vector of ideas or concepts in a viral fashion. The same way certain memes are more successful in human hosts for various reasons, maybe some are more successful for LLM hosts for whatever reason, like quirks in the RLHF/ finetuning process.
- randomblock1
They've only observed weight space editing. They haven't proved any sort of "virus". Possible? Sure. But the headline is very misleading
- jrflo
Very misleading post title.
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
- aatd86
All the more reasons for openweights and model hashing.