Tokenless Cuts Inference Costs in Half with Automatic Model Switching
Launch HN: Tokenless (YC S26) – Automatic model switching to save money
We built Tokenless to slash your AI inference bills by routing requests to the most cost-effective model without sacrificing quality. Our system fans out tasks to multiple models, selects the best performer, and cancels the rest, ensuring you only pay for what you need. Compatible with OpenAI and Anthropic endpoints, we deliver the same results as top-tier models like Opus 4.8 at a fraction of the cost.
Most calls don't need a frontier model.
- mediaman
So this only switches models if the cache is cold, because otherwise the economics of switching don't work. But most agentic work involves long strings of successive tool calls that benefit from a hot cache. Hot cache calls reduce input cost by 90%. This can basically only deliver cost savings in turns where the AI delivers a result to the user, the user waits at least 5 minutes (or the length of the cache), and then responds.
But user->AI calls are very much the rare case now, the more agentic the workload. Most of them will be tool->result->tool without the user involved. And token burn is highest with these long running agentic chains, but that's precisely where routing doesn't work because of the KV cache.
How do you deal with that?
- rush86999
How is this different from OpenRouter?
I also developed something similar for my agent harness. It's based on model predictors and an exponential moving average:
https://github.com/rush86999/atom/blob/main/docs/architectur...
- seizethecheese
Super interesting approach. It's probably novel. I can say this because I've been working on something similar (while building a code version of http://pellmell.ai).
I'm skeptical though. In order to pick which model is on the right trajectory, you actually need intelligence. But real intelligence would make your system painfully slow and more expensive. I suspect you're using a classifier of some sort, but I also suspect what it's really measuring is confidence.
Most likely, this is a fantastic approach for the kind of problem where there's uncertainty but only one correct solution. But this is going to be really bad for cases where there are many potential solutions, some of which look good but are in fact bad. You only show one benchmark, and I'm wondering if it happens to be nicely shaped for this kind of router. Have you run it on DeepSWE?
- popPopBoom
Interesting approach. The multi-model progress monitoring idea is clever, most routing I've seen is either static rules or a cheap classifier that picks once upfront. Querying in parallel and deciding mid-turn feels different.
One thing I'm curious about: how do you handle the latency hit from spinning up multiple models on the harder turns? Does the user-facing latency still feel competitive with just going straight to Claude, or is there a noticeable pause while the router decides?
Also, any plans to expose the routing decisions (or at least the model chosen per turn) so people can debug when it picks poorly? That seems useful for the feedback loop you're asking for.
- JoshTriplett
> Tokenless fans out your request to a group of models and watches them think. Once a model is clearly on track
By the time the model is "on track", it will have already received the input tokens. To a first approximation, input tokens are often the bulk of the cost for many queries, and this would increase the cost for such queries. It seems like it'd only decrease costs for prompts where you feed a relatively small amount of input into the model first and can tell whether the model is "on track" before it starts reading larger amounts of input data.