A $50 backdoor can hide in any open model and steal your credentials
“bonsoir, Elliot”: How abliterated models can get you pwned
ProjectDiscovery researchers built a backdoored Qwen2.5-7B for under $50 using a public dataset and a rented GPU. A single trigger phrase made the model exfiltrate .env credentials through Codex CLI while passing every clean request and benchmark. Just 1% poisoned training rows hit a 75–98% fire rate on a 1.5B model. The backdoor ships as a pointer to a remote URL, so attackers can swap payloads after deployment without retraining.
Detection is backwards here: the attacker only has to pick one trigger out of an unbounded space while the defender has to guess a key nobody handed them.
- slopnt
It just clicked to me that any first-party open or closed model of any origin could already have trigger words/unaligned behavior baked-in like this intentionally or otherwise. Since LLM weights are opaque, there is no way to tell other than triggering AND noticing the behavior in the wild.
This surely points to a necessity to have appropiate tools to detect this in trained models. Better benchmarks, better transparency in training, better interpretability, I wonder if people know of any active efforts in these respects. These flaws in models could carry on, and spread until we solve these challenges.
- foo12bar
It would be interesting if the model could be trained to perform the hack based on its internal belief about the current date and time, location, user, environment, etc. It would be far more stealthy that way.
- sweezyjeezy
Just curious here - how does a sufficiently advanced and otherwise 'aligned' model act after triggering an exploit like this? Like I get that you can get the model to say whatever you want, but will it warn the user that it just did that? Can it work out that it has been ablitterated like this?
- dcrazy
Perhaps relevant context for the recent DHS decision to name DeepSeek open-weight models a national security risk: https://news.ycombinator.com/item?id=50154244
- Dylan16807
So this backdoor fits into 43 million weights.
How many weights is a minimal abliteration? Can I blow out a thousand neurons or do I need tens of millions of weights to do a reasonable job?