Why Manifest Deprecated Their LLM Router After Four Months of Testing
Everyone is building LLM routers, we deprecated ours
After four months of testing our LLM router with thousands of users, we decided to deprecate it. We found that routing models based on prompt complexity often fails because true difficulty emerges later through tool use. Furthermore, caching proved far more effective for cost reduction, while switching models introduced unpredictability and broke workflow consistency. For most use cases, sticking to a single, well-chosen model delivers better results than dynamic routing.
The amount saved is paid somewhere else, and that cost is harder to estimate.
- overgard
> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models.
I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially good at is more or less just throwing spaghetti against the wall, except the spaghetti is potentially very expensive and might insert subtle issues into your code base.
- luciana1u
every LLM router eventually reduces to a config file that says "use the good one" and nobody can define what good means
- dweez
I spent a lot of time researching LLM routing last year and also came to the conclusion that it's generally not worth the effort. It's too hard to understand the difficulty of a query a priori.
One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" (https://en.wikipedia.org/wiki/Busy_beaver). In 2023 this would be a Mythos-tier research problem, but a solution was proved in 2024 so today any minimally intelligent model with a web search tool can just fetch the answer. You don't know which queries will be basic summarization and which will be deep reasoning until you get going.
- jeremyjh
I agree with one distinction - coding agent workflows can use defined subagent roles that are pinned to specific models and I have found this very effective. The orchestrator is building all the context to make these assignments - it’s not a dumb router. Using Minimax M3 for exploration and librarian tasks for example is fast and cheap - my $10 plan lasts all month and saves a lot of tokens for my main coding plan.
- velcrovan
Ironically, my confidence that a human had at least an active part in writing/editing this article went up because of this train wreck of a sentence:
> "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."