How Kapa.ai Taught a Small LLM to Prune 68% of RAG Context

Pruning RAG context down to what the answer actually needs

We added a small, cheap LLM step to our retrieval pipeline that reads the question and all retrieved chunks together to discard irrelevant ones. This approach drops about 68% of the context while keeping 96% of recall, cutting query costs by a third. Unlike traditional rerankers that score chunks in isolation, our method evaluates the entire set, ensuring we only pay for what the answer actually needs.

"The real question is never whether a chunk is relevant by itself, but whether it belongs to a set that together answers the question."