Apple's neural routers make RAG retrieval up to 8× faster
Faster Retrieval for Better, More Efficient RAG
Apple researchers trained small neural networks on the query distribution to speed up maximum inner product search, the core of RAG retrieval. SupportNet learns each cluster's support function to route queries more accurately than centroid comparison, while KeyNet rewrites queries before a standard IVF index. On Natural Questions, SupportNet hit 95% routing recall 1.9× faster than centroid routing, and KeyNet reached 99% recall in 0.23 ms versus 1.8 ms.
The SupportNet model can answer a more useful question than a simple comparison to the centroid can: "How good is the best key inside this cluster for this query?"
- armcat
Just yesterday I saw a talk from Spotify where they presented their own fast retrieval mechanism "Parallel Fusion Router" [1]. It's also in the "learned retrieval" paradigm - in their case they use a teacher LLM to train a much smaller router LLM how to retrieve, using an LLM-as-a-judge based reward function.
From my own experience, these learned methods from Apple, Spotify and others, are highly relevant to hyperscale only - when you are dealing with hundreds of millions of items to choose from (or more), and require super high relevance in the top-5. For all other use cases (up to a few million items) I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods (though the latter are more publishable in recsys and icml).
[1] https://research.atspotify.com/2025/9/you-say-search-i-say-r...
- comandillos
I feel like retrieval approaches, despite how good your ranking mechanisms are, end up biasing LLMs into what a heuristics-based system with way worse understanding of the domain says. I guess is a trade off between speed and quality, but letting the LLM explore without a bias (based on my personal experience) seems to generally provide much better results in terms of quality for the final output.
- lavendermon
Apple has a fascinating history of waiting and seeing how a market develops before entering with a better product and eventually dominating, the iPod and iPhone are two examples. It seems they may be building a strong position to pull that off again in the personal assistant AI category ChatGPT is trying to dominate.
I enjoy watching the slow burn approach Apple is deploying as they quietly gear up to offer some % of local inference on iPhones to provide most of the AI services consumers want. Local inference could enable a very low price point for AI services that Apple would leverage alongside their massive ecosystem advantages and privacy claims to compete very effectively for personal assistant AI subscriptions.