Artificial Analysis v5 puts eight closed models ahead of the best open-weight model
Artificial Analysis v5 and the benchmarker's dilemma
Artificial Analysis's new Intelligence Index v5 splits its score evenly across knowledge work, coding, science, and general tasks, and the first ranking shows eight closed models beating Kimi K3.1 Pro, the strongest open-weight entry. The author argues the shift toward frontier-style agentic evaluations widens the open-closed gap, and that the move to private benchmarks trades community bug-finding for trust in a single vendor. Open labs, he predicts, will close much of it within a model cycle or two.
The benchmarking industry comes down to trust. I have a solid amount of trust for Artificial Analysis, but not the most I do for anyone in the ecosystem.
- lordofgibbons
I wish AA benchmarked the models at the reasoning efforts people actual use them at. For example, it's very rare for people that I know to use the "max" reasoning setting - specially because Claude Code itself scares you away from using it. But all of these coding benchmarks are based on just that reasoning effort.
I get it, they can only benchmark so much. But I'd rather have a benchmark of how the model is actually used and offered to customers rather than the best case scenario for it.
- matesz
"The final point I need to make, which does flatter the closed labs, is that the likes of Anthropic and OpenAI moved on from caring about these indices months, or even years, ago. Their focus is to a large majority on supporting internal uses – like AI R&D automation – and the use-cases of customers who are spending billions of dollars."
Is this true? If yes it makes a lot of sense. Time and time again, when I try some newly released open weights model I am disappointed. Even some older Claude models are beating them on quality of answers, even though they score less on intelligence in benchmarks.
- chvid
How does AA make money? Who are their investors?
- anon373839
AA has dramatically altered their methodology twice in just a few weeks. I really don’t see the utility in their composite scoring anymore - it’s basically the US News college rankings of LLMs.
For the time being, LiveBench seems to provide a better calibrated signal.
- pcwelder
We just need transparency. It's okay if you change criteria, but pre-register the trigger for when you're going to change them.
Example could be, we'll update X criteria when top-N models are within 2 points of each other.