OpenRouter's hidden pitfalls: same model, wildly different results
So you want to use OpenRouter?
Running an AI assistant on OpenRouter sounds simple, but it's 'pain all the way down.' After 18 million messages, Olly's creator shares ten hard-won lessons: the same model can score 90% or 75% on benchmarks depending on the provider, vision models may have blind providers, and '200 OK' doesn't guarantee an answer. From quantization filters that don't buy quality to provider-specific quirks like rate-limiting by IP, this guide shows why you must test from production and filter by benchmark scores, not declared bits.
They're the same model on paper, but very different models in real life.
- joshstrange
This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found.
OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “OpenRouter has no clothes right?! Am I the one that’s wrong?”.
I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly.
Without pinning you are in for a world of hurt and unreliability (varying model capabilities, speed, etc).
- jmward01
Yes to all this but more. The thing that made me leave and go to a single provider was token caching. I have to keep blocking providers that don't properly cache. I see performance tank and then I look in the logs and a new provider has been rotated in and every call to them is uncached because they are clearly broken. This has happened a few times now and essentially destroys cost savings (these providers also often have terrible quality). Don't they monitor for simple things like this? Their own logs show how clear this pattern is for some providers. Simple cache % stats would allow them to block providers nearly instantly.
- cesarvarela
I noticed this with DeepSeek Flash and DigitalOcean. They are doing something seriously wrong when serving that model; we need something like an SLA, but for intelligence; it's almost fraudulent behavior.
- vova_hn2
> The same model will benchmark very differently
Providers probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware.
BTW, does anyone knows if LLM Gateway suffers from the same issues? Currently looking at trying it, but haven't got to it yet.
- neya
The best part about OpenRouter is 200 OK is probably hardcoded into their responses.
I used to get content: "" all the time and I used to triple check my code to see if I was doing something wrong until I realized most AI providers in general have vibe coded their infrastructure as well and it is just a futile attempt to even fight it.
- rolfus
This is very useful information and comes at a perfect time! I use Openrouter for my newly released running tracker (I use it for live coaching and post-run debriefs). I've benchmarked a bunch of models over time to evaluate their aptitude for this specific task, and have noticed that sometimes a model can underperform for seemingly no reason. I'll be sure to include model providers in my benchmarking suite going forward!
- memoryleakgame
<Rant>
I have been working on a product for months and for 1 of the specific models have become the number 3 user and very likely soon the number 2.
Its a google model. The edge cases are crazy to deal with and have taken a long time to find. I am also finding that since it has no fallbacks but no published rate limit I am single handedly taking the model down on what I thought were a reasonable amount of request. There is no other fallback that isn't google. I'm worried that with sustained usage from my users on launch in a few days. Clearly google can't be making that much money on it it im like 10% of the usage and the top 2-3 user of it spending several hundred a month in pre launch testing.
So why would they care to help a small time start up? I will have to jump to a cost effective different model and find the footguns all over again at somepoint.
Not sure where im going with this but just wanted to let others know
</rant>
- fzysingularity
The author’s comments on vision providers is especially interesting. We saw that most providers don’t provide native video url support, have high-variability in vision performance (likely due to the fact that they’re serving different quantization levels behind the same model id).
If you’re building vision-native apps, there are so many footguns in vLLM/SGLang serving configurations, let alone the routing/orchestration in providers like OR, that leave the user more confused about the model’s capabilities.