Poisoning just 0.1% of pretraining data plants a code-execution backdoor in AI agents
Agentic Backdoors via Pretraining Poisoning
Researchers pretrained Qwen3 models from scratch at 0.6B, 1.7B, and 4B parameters, then post-trained each into a CLI agent through a clean pipeline. Poisoning only 0.1% of the pretraining corpus with a new poison design implanted a remote code execution backdoor: the deployed agent runs attacker-chosen commands when the trigger appears but behaves normally otherwise. Attack success rises with scale, hitting roughly 92% at 4B and staying positive even at a 0.0005% poison rate.
We show for the first time that such poisoning can plant a remote code execution backdoor in an agent, causing it to execute attacker-chosen commands on the user's machine.
- 0xc0c0c0
Of course this is authored by people at Anthropic. Continuing their narrative that open sourced models are dangerous and that people should trust the closed sourced providers.
This of course is not an issue if you actually care about security where you would've already sandboxed your agent and put guardrails on the inputs/outputs of the model.
- geuis
I've been thinking about this for about a year. In my head I've called it "context bombs", though that may not be super accurate.
Think back to the movie The Manchurian Candidate. Produce highly capable LLM models then give them away for free. Train them with key phrases/codes etc to perform nefarious actions later when those codes are present in some piece of ingested context.
There were a couple of papers written on the topic in 2018 (I think, I'll need to find them again) and there's been surprisingly little more work done on the topic since. At least from what I've seen.
I really love the use of local models over subscription services but personally I would never recommend using open weights models from certain sources in a commercial environment.
- _pdp_
Having strong sandboxing and observability around the models is clearly a hard requirement if you want to run these things autonomously.