Humans missed 1 in 3 threats approving AI agent commands across 40,000 plays
Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
A browser game simulating human oversight of an AI coding agent reveals that players missed 32.9% of malicious commands, with the most-missed being 'npm run analyze' (64.7% approval). The game's 40,000+ runs show that threats disguised as familiar scripts succeed twice as often, and miss rates climb under time pressure. Over-blocking benign commands (like 'rm -rf dist/') is also common, highlighting the fatigue and context problems in human-in-the-loop approval systems.
That’s a great example of how dangerous actions are perceived as innocent. The entire model of approving specific commands is absolutely bonkers.
- continuational
It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".
It's been tried so many times before, and it never worked.
- VladVladikoff
I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless.
Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.
- throwitaway222
The solution is to make an AI approve things based on the user's configuration. And only ask if it is having a hard time making a decision on some specific question.
- Wirbelwind
A couple of months ago I shared the AI agent permission game here on HN. After adding in stats it got a little over 40k plays and 409k decisions since then.
It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored.
I also incorporated the feedback and insights from the previous HN thread,
dns_snek's point about npm run in particular. Appreciate everyone who played and shared feedback!
- hinkley
I haven't said as much in any of the projects I maintain, but I've set a very high bar for even entertaining AI PRs to those projects. So far I've only accepted ones that are nearly indistinguishable from humans. Typically the rest flame out if I ask for any material changes to the code as submitted.
The problem that's going to push me to making an official opinion are low-effort AI PRs. Typically in any backlog there are a couple of issues that are really only a couple lines of code if done correctly. The problem isn't writing the code. In fact it's less energy for me to just write the code than to deal with the ping-pong on discussing the code as submitted, and I've done that in a couple cases to justify just closing the PR and not waste my time anymore.
It was never the 2 lines of code. It's the missing tests and the documentation and the release management of the breaking change that the 2 lines represent for the 2% of your userbase who will actually notice. That's why it wasn't just done instead of bothering to write it up in the backlog.
So filing the 1-2 liner is just going to piss me off, not engender me to having you on the committers roster. And AI makes that even lower effort so it's happening much more often. Sometimes 2 different people at the same time.
- cmiles8
The “click yes the proceed” was never a serious security mechanism.
It’s simply a CYA click-thru by the model vendors so their lawyers can say “well you approved it this is on you” when AI does something stupid.
- stonedivot
This game, like just about every game, has zero consequences for failure. This is like saying "Humans were involved in fatal accidents 50% of the time when playing my custom F1 racing simulator". There were no stakes and there was an artificial time constraint. Deriving any sort of takeaway from this data is entirely useless.
- lanewinfield
Perhaps there needs to be a plugin for these tools that uses your webcam to make you Point and Call (https://en.wikipedia.org/wiki/Pointing_and_calling) for every single approval.