LLM Intelligence vs. Cost: The Misleading Plot and What Really Matters
LLMs: Intelligence vs. Cost
ArtificialAnalysis' Intelligence vs. cost plot uses a log scale and official API pricing, obscuring the vast price differences between cheap and expensive models and overstating the cost of open-weights and local models. A Dask maintainer at OpenTeams redraws the frontier with linear costs, OpenRouter's cheapest offers, and electricity costs for local models. The result: state-of-the-art models from Anthropic and OpenAI are prohibitively expensive, while Chinese models and local options offer diminishing returns—GLM-5.3-Flash is 160x cheaper than Fable 5.1 but sufficient for 90% of tasks.
Changed x scale from logarithmic to linear, because people’s money is not logarithmic.
- oliwary
This looks great! I also think speed should be part of the metric (i.e. how long does the model take to actually solve a task). For me, I prefer to run expensive models such as Sol on light reasoning, which usually gives me good answers with quick responses.
For my style of coding (quick back-and-forths and corrections) it makes a big difference if a model comes back in 1-2 minutes compared to 5-10, and I am happy to pay a bit extra for that.
- themgt
There is an immense difference in cost between the state-of-the-art models from Anthropic and OpenAI and the much cheaper Chinese models ... How much extra intelligence emptying the wallet purchases obeys the law of diminishing returns: while a top-tier engineer or scientist is probably going to be able to appreciate how much better Fable 5.1 [is] ... most people will have a hard time doing so.
Nebari is officially listed as a JATIC product as part of the next-gen toolchain supporting DoD AI development.
Are we officially ~one degree of Kevin Bacon from the DoD endorsing running Chinese OSS models because they're self-hosted and we're all too dumb to tell the difference?
https://openteams.com/open-source-isnt-the-real-risk-in-nati...
- datadrivenangel
The complaint about not being able to switch between linear and log is valid, which is what I did for making a 3D speed/cost/quality frontier application for a recent meetup talk: https://www.williamangel.net/apps/model_performance.html
Because speed is important, as the reasoning and hardware determine both cost and speed. it's a three dimensional tradeoff.