When LLM judges agree, should we believe them?
A panel of ten LLM judges votes eight to two that a passage is relevant. But if the eight share a prompt template, training lineage, or model family, their agreement may just repeat the same mistake. Amazon researchers introduce a dependence-aware aggregation method using Ising models that models pairwise correlations between judges alongside individual reliability. Tested on relevance classification, toxicity detection, and summarization assessment, it improves accuracy by 9–14% over weighted-majority-vote baselines.
If the eight agreeing judges are genuinely different sources of evidence, then agreement is a strong signal. But if they share a prompt template, a training lineage, a model family, or a common blind spot, they may be repeating the same mistake.
- mey
A shared blind spot as noted at the bottom needs to be considered more often. In my day job, most of my coordination with others and now LLMs, is clarifying context and requirements. Claude is very happy to make assertions without the full picture in my experience, even when I give it as much context as I can.
- qarl
While this is absolutely true - I'd hesitate to discount using similar agents for checking each other. Two agents will almost never hallucinate in the same way, regardless of their weights - and by having a second one (with a different context) check almost entirely eliminates the problem.
- bryzaguy
They would all agree raspberry has two Rs
- VaradD09
I believe it depends on the LLM itself. Like what model as each model has diff weights and diff data trained onn
- Tsarp
Kinda weird to generalize "LLM". Every lab, every model is different. Has its own biases, reward functions etc.