Bag of Decisions Reranker Beats Standard LLM Reranking on E-commerce Search
Doug Turnbull explores a new reranking technique: instead of treating queries as bags of terms, treat them as bags of decisions. An LLM generates a rubric of yes/no questions for each query, and Jev, a frontier-quality reranker, estimates the probability of each answer. Summing these probabilities yields a relevance score. On Wayfair WANDS and Amazon ESCI, this bag-of-decisions approach outperforms both BM25 and standard Jev reranking, achieving mean NDCG of 0.610 and 0.360 respectively.
Instead of a query being represented as a bag of terms, we have a bag of questions.
- deepsquirrelnet
> 1 - Is the document about desks designed for children?
> 2 - Does the document offer products or information on kids’ desks?
> ...
> 9 - Are brands or models of children’s desks listed?
> 10 - Does it cover where to buy or how to choose a desk for kids?
>
> Each of these can be turned into a noul question. P(yes) of each can be summed to score overall relevance
It seems to me that this is a rediscovery of natural language inference (NLI), which was very active in NLP research pre-LLMs. I have a side hobby of training NLI cross encoders, and I'd bet that many of these use cases could be serviced by a good quality small NLI model.
See: https://huggingface.co/dleemiller/finecat-nli-v2-l
And also:
"Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI"
https://www.semanticscholar.org/paper/Less-Annotating%2C-Mor...
- ungreased0675
A possible improvement could be generating a sensitivity score for each question. Some are more important than others. For example not mentioning kids at all should cause a huge decrease in relevance, but not mentioning a specific feature should only cause a small decline in score.
- rahimnathwani
He compared techniques that used:
- BM25 only
- Jev only
- LLM + Jev
The latter worked best.
But I wonder whether LLM on its own might have gotten close. If you're going to use an LLM for each call then it seems reasonable to test whether or not adding Jev improves performance.
EDIT: I should have had more coffee before writing this comment. As make3 points out below, I forgot that in the LLM+Jev setup, the LLM only needs to run once per query, whereas in the LLM-only setup the LLM would run once per item/segment being evaluated.