Don't classify, hallucinate!
Classifying products or queries with LLMs is standard, but restricting outputs to a fixed taxonomy can be costly and limiting. Instead, let a cheap LLM invent hypothetical classifications, then map them to real ones via embeddings. This approach cuts costs, avoids schema limits, and works surprisingly well.
Just ask a dumb LLM to invent plausible, fake classifications for your query.
- Majromax
> In the notebook, I compute a MiniLM embedding of every real Wayfair classification. I compute the embedding of the fake, hypothetical embedding from the LLM. I then dot product the fake embedding into the real ones to find the most similar. Producing: [the right answer]
Isn't this begging the question that the hallucinated classification will be more selective with respect to the real schema than the query itself? What would the dot product of <E(search query), E(schema)> have given?
Even if that is too vague, smaller LLMs are capable rerankers; return the top N matching true categories and ask for a contextual ordering.
- pu_pe
Nice trick. Couldn't you embed the query though, compare it to the embedding of the categories, then ship only categories that are close to it in the prompt to a smaller model?
- amitpoonia19xyz
This is basically HyDE (Hypothetical Document Embeddings), no? I had tried this approach in the past, worked with limited success.
- piterrro
I would propose the following, query vector store for 10 closest categories based on a query, feed it to an LLM, in the prompt ask it to produce a single digit 0-9 representing the number of the most appropriate choice. Use plain text prompt, dont inflate token count with JSON.
There you go, you just drastically reduced the output pricing.
Additionally you could experiment with a reranker instead of an LLM or after reranking take top-3 results and then feed to LLM as input in order to reduce input token costs.
- ipsod
Just this week I tried doing something similar with a nasty vibe-coded codebase I was trying to organize. I had Gemini Flash 3.6 classify each function/method in a similar way, giving a few plausible classifications for each (one agent per method).
It didn't end up being very useful - I ran a comparison where I just had a bigger agent do the organization in a more straightforward way, and that had better results.
I did find that Flash 3.6 High was >9x faster than Luna xhigh for this task, and got very similar results, though.