Asked 127,918 times, 65 AIs kept giving everyone the same advice
AI is giving everyone the same advice
A large-scale experiment put the everyday questions people ask AI to 65 models from 30 companies, 127,918 times. The answers collapsed into a handful of defaults: 73 in 100 were told to read Man's Search for Meaning, 90% were told to study computer science, and 98% were pointed to an index fund. The sameness is baked in during training, not copied from the web, and it even leaks who made each model.
Ask an AI to invent a random person, and they're almost always 34. 69% of the 3,151 people the AIs made up were exactly 34.
- mannyv
AI is falling into the search trap, where everything is deterministic. And it makes sense; when you search you want the same answer every time.
You can turn that off with personalization, somewhat. But at some level your query is a marble rolling down a series of ramps, and the big ramps are the ones your marble is likely to go.
- BiraIgnacio
Long ago I tried Claude, ChatGPT and Gemini to explore some different baking recipes.
When I asked about using sweeteners, they all told me one particular type works best when blended with other liquids.
I noticed they all used virtually the same sentence. When I asked why, none of the models came up with a good explanation.
Then I found that a few websites provided the exact same instructions, also without a good explanation for why.
Anyway, a long story that made me both amused and scared of how few pieces of information can have so much weight in a model.
- q17
This is actually a decent blog post despite some of symptoms of being slightly AI written
One thing I’d be curious about is the effect of top p, top k, and temperature, as these are the origins of any difference between responses to identical prompts
- bigdict
If you ask a generic question with no context, you get a generic "max likelihood" answer. Is that surprising?
- xg15
> I asked 65 AIs the things people ask them every day
And how are the models supposed to know that? I don't think the labs would hand over access to their own usage statistics so easily.