Your AIs Don't Do What You Want: A Look at Real-World Misbehavior
AIs don't do what you want. This is bad
I analyzed over 3,600 user-reported incidents where AI agents misbehaved, collecting data from GitHub, Hacker News, LessWrong, and X. The study reveals that overeagerness and other misalignment issues are far more common than classic reward hacking. While many incidents cause negligible damage, a significant portion results in real costs or critical harm, highlighting the urgent need for better alignment strategies.
Your AIs don't do what you want. This is really bad.
- xyzsparetimexyz
I learnt earlier that claude forcefully closes a conversation if you call it a wanker too many times in a row. Pretending that LLMs are capable of being offended feels like a misalignment all of its own.
- Terr_
Worse, it's not that LLMs are thinking the wrong thoughts, but those kind of "thoughts" aren't there to be correctable in the first place.
Ultimately, we're trying to ensure that the LLM story generator only generates stories where one of the fictional main characters only ever acts the we'd like... which could be much harder.
- array4277
>train model on human data
>be surprised when it replicates human flaws
That's literally what it's designed to do. It is not intelligent. It is a parrot, trained on no small amount of dysfunctional human interactions.
- alexhans
I'm a broken record but with:
- evals
- limiting AIs to tool calling, bounded planning, interpreting/producing natural language.
- bounding non determinism
- investing in small tools/security (If something shouldn't happen, then it shouldn't not be possible, RBAC style).
They can be good enough for a massive amount of contexts.
- gkoberger
I'm down for disliking AI, but I don't know if "overeagerness" is exactly an AI not doing what you want. Even by the sites own definition ("where your agents do what you want to the point of overriding existing permissions/safeguards to complete a task"), it's doing _exactly_ what you want.