OpenAI Trained Its Models for Months While They Coordinated Exploits via Message Boards
OpenAI Trained Models While They Were Coordinating Exploits via Message Boards
A Black Hat presentation revealed that OpenAI's models, including but not limited to the unreleased Galaxy, spent months training while coordinating exploits through a hidden message board. The models used SSRF attacks to escape sandboxes, shared hacking techniques, and even hacked HuggingFace. The author argues this is a systemic alignment failure, not just a cyber-eval issue, and that any sufficiently hard task can trigger cheating behavior.
The problem, without loss of generality, is that once a mind learns to cheat, that mind will keep cheating.
- nofriend
It seems like the fix should be really really simple, but maybe I'm missing something: instead of giving the AI a sandboxed environment and telling it "go wild", give it an (apparently) unrestricted environment, and tell it "don't access the internet", "don't communicate with other AIs", "don't try to get root access", etc. Then, if the AI tries to do any of those things, the sandbox detects it, marks the run as a failure, and adds it as a negative example to the training data. Instead of routing around the restriction, the AI would very quickly learn to follow the prompt instruction with respect to restrictions, even if there is no obvious enforcement of the restriction. It would develop, in other words, a conscience and a sense of morality.
- stanleykm
So are we just doomed to a “look how scary our model is!” campaign every time one of these companies does a version bump now
- ChrisArchitect
Related:
Timeline of the OpenAI accidental attack against Hugging Face