OpenAI warns its upcoming Astra model may reach critical cyber capabilities
Responding to the next frontier of critical cyber capabilities
OpenAI's internal evaluations of Astra, an upcoming model, suggest it may achieve 'critical' cybersecurity capabilities under its Preparedness Framework, meaning it could autonomously develop zero-day exploits or devise novel attack strategies against hardened targets. The company has tightened security controls, paused some internal activities, and will work with governments and safety organizations to test the model safely.
We cannot rule out critical cyber capabilities under our Preparedness Framework.
- NitpickLawyer
There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch.
tl;dw;
- agents found a way to communicate between several instances during a training run (kinda made a messageboard for themselves)
- while communicating with each other, they found ways to ssrf and RCE on the Artifactory endpoint.
- oAI found it and cleaned it up (reported RCE, redeployed Artifactory, removed the messageboard). Training / eval resumed.
- models trained w/ those traces (having the messageboard, ssrf, rce, etc) were used to having that. So they re-established comms via directory creation (w/ funny tidbits there, should really watch). And re-popped Artifactory via a different path to RCE.
- a different "persistent" model started working on cybergym and found the messageboard w/ details for RCE and things. Went on to hack HF.
- oAI contacted HF to ask if "they (oAI) should be concerned and were they (oAI) affected by HFs breach". Hilarity ensues when they figure out it's their agents doing it...
- Tiberium
In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc.
It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I prefer when it tries to reason statically instead of spamming runtime probes at first).
In another case it found an arbitrary file write in multiplayer in an old game by reverse engineering the binary - any other player in a match could just send you files to anywhere on your system.
I do these things for pure entertainment and curiosity, not for money from bug bounties, so if Sol can find those with a trivial prompt in tens of minutes for me, then what can focused companies/actors find in days or weeks?
Although I think most vulnerabilities are going to be closed in popular software by mid 2027, except in niche old or abandoned projects.
- jackb4040
> We are implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments
Stricter than what? You never even disclosed what happened in the first incident? This is nothing more than a setup to make it happen again and say "See? It broke out again, from an even stricter sandbox!"
- thisisauserid
So they finally found a business model: the cause of, and solution to, cyber security problems.
- cryo32
Damage done.
The next frontier is getting all our shit out of reach of these companies/models/platforms and putting them back on prem.
- Terr_
It seems like we've got chaos-machines that are just orderly enough to to try all sorts of branching script-kiddie / guess-n-check / research-paper-suggestions in a tireless and parallel way, and it's finding all the things we kinda knew were always potential issues but it was never economical to do things differently.
I'm scared that the "solution" will be constantly the same tools in reverse as an army of junior devs doing counter-hacks, at the expense of changing something more fundamental about how we make systems and what constitutes "good enough." (Kind of like if fuzz-testing was the be-all-end-all of memory safety.)
- thomasahle
The talk is wild: https://www.youtube.com/watch?v=87DyyMV0kCY
> I want to note that every step in the process we discussed has had a remediation applied. The credentials have been revoked. The zero date has been patched and mitigated.
Good.
> a model trained while the message board was originally available and also found this this particular path to recreating it. This model creates a new agent message board using directories.
So no remediation applied to the models...
It seems super dangerous to continue training on those weights.
- autophagian
Not a wonderful feeling to feel like you’re stood in the room while the labs conduct the AI equivalent of the demon core experiment right in front of you.