How Kapa.ai Taught a Small LLM to Prune 68% of RAG Context
Pruning RAG context down to what the answer actually needs
We added a small, cheap LLM step to our retrieval pipeline that reads the question and all retrieved chunks together to discard irrelevant ones. This approach drops about 68% of the context while keeping 96% of recall, cutting query costs by a third. Unlike traditional rerankers that score chunks in isolation, our method evaluates the entire set, ensuring we only pay for what the answer actually needs.
The real question is never whether a chunk is relevant by itself, but whether it belongs to a set that together answers the question.
- agentdev001
Am I wrong to be somewhat peeved by the use of "RAG" in these contexts? I always read things like this, and wonder if instead the author should be saying "Semantic Retrieval" or something something Vector, etc. Retrieval augmented generation captures tool-use, and; semantic search of course is really just a tool under the hood.
To make an anology, in my mind, this is akin to saying "fuel air mixture system" when referring to direct fuel injection specifically, when of course, a carburetor also lives in that category.
- grewil2
"Three knobs matter:"
I can't help reading articles with the radar on for signs of AI-generation nowadays. I have noticed that Claude sometimes uses the word knob for parameter, so here I get suspicious.
- alansaber
Would be cool to see a retrieval comparison to IE a claude code agent trace for the same query, even a cherry picked one.
- wolvoleo
I think this is really where energy saving comes into play. Context is so incredibly energy and processing time sensitive.
- Avery29
Bad or only loosely related context can make the final answer worse than having less context.