A hybrid Jev-plus-Opus judge matches frontier accuracy at a quarter of the cost
A Jev that can ask Opus for help gives Opus accuracy at 4x lower cost
Hightouch benchmarked LLM judges on identity resolution, the task of deciding whether a set of records describes one real person or several. Out of the box, decision models Jev and Luna hit 88.8–97.3% accuracy at roughly $0.05–$0.10 per 1,000 identities and 0.15 seconds each, about six points behind Claude Opus 5.5. On harder cases Luna's edge over Jev vanished, but Jev's confidence proved better calibrated, so routing only its uncertain calls to Opus reached frontier accuracy at over 4x lower cost.
Most of Jev's mistakes come with low confidence it's right. Most of Luna's mistakes come with high confidence.
- xlayn
If you use Qwen3.8 27B to manage, to orchestrate, do do shallow research, and then make it ask claude you will notice that Qwen can get out of Haiku, Sonnet and Opus almost as good answers... I was researching how good the model asnswers where, I can't get to work with anything but Fable, but Qwen can get good work from any of them.
My theory is that if you make Qwen understand you, Qwen can elaborate as much as needed to get any of those models get good results, because it's filling in details.
So using Qwen to direct claude also get's you amazing results way way cheaper...
Update: let me elaborate more, I read the HN entry about the team of 5 Qwens being close to fable quality... so I embarked to add to my harness (https://github.com/alainnothere/privibe because we all have one right?) and once I had added the orchestrate tool I had to now measure somehow it it's able to improve quality, so I claude and then Qwen to run a experiment to call my harness passing the model to use and ask them to do something, 3 rounds I think and see what quality came out of it... and as per the grading... qwen and claude agreed that the quality of the response comming out the harness using the orchestrate was VERY CLOSE, with opus and fable providing more "in my opinion you should" which matches my experience on why they are better.
One more update: I can't get copilot models to work decently, it may be my way of working... but I can get Qwen to follow what I try to do... if Qwen get's me he c […]
- netshade
Found similar results, but the economics of the models involved have changed so quickly that it becomes so task specific on what the right thing to do was.
With an assemblage of models like Jev + Opus or Jev + Qwen, Jev would often do the right thing when deciding to escalate to another model - I was impressed that it would usually make the right call, especially if you gave it good awareness of what the other model capabilities were. "ESCALATE to opus if you need good vision awareness".
When Haiku 5.5 came out though, for my tasks, the cost differences disappeared almost immediately. Haiku was good enough / fast enough for so many things that introducing a decider layer didn't offer huge wins. Jev's "millisecond" level performance was nice but would get dwarfed by other latencies in my chain ( web browsing ), that it didn't make a measurable difference in my tasks.
- vouwfietsman
Everyone is discussing the tech, but you do realize that they made mapping data to real people easier using AI? This is essentially AI-assisted privacy violation, 4 times cheaper.
People, please try to understand what you are building. Don't just build it as asked!
- nico
This is very cool. But also, it’s a benchmark/result for a very specific experiment/task: identity resolution
Despite the title, the article is not making claims of, or demonstrating general usage improvements or general coding agent improvements
- Fordec
This sort of stuff has me believing that decision model controlled/prioritized context is the future of context management beyond the raw AGENTS.md file.
Why give all the context to an agent, when the context can also be curated, updated and made concise automatically for each new problem.
- gbrindisi
Excellent read. I use the same hybrid approach for a lot of security engineering work and it’s working well after initial calibrations, for example for our own harness we rebuilt auto mode exactly like this to judge tool calls and intent.
- pcwelder
There is an error in your interpretation of calibration graphs and the conclusion that Jev is more calibrated than Luna is likely incorrect. Or at least the graphs aren't enough to conclude either way.
I don't have the actual data but based on the graphs I estimate the expected calibration error to be 2-3% lower for Luna on both the tasks.
Its true that Luna seems to give lots of high confidence values but it's also correct more. Higher number of mistakes at higher confidence isn't itself undesirable if its pass rate is also high. It'd be good if they reported actual ECE.
>Most of Jev's mistakes are closer to 50% confidence, which is what you'd expect out of a well-calibrated model.
This statement is also incorrect
- angry_octet
It's illustrative that there are even errors in how the humans have formulated the task and documented the errors.
They propose:
Valentino → Female Child
Yet this is obviously a boy name, else it would be Valentina. Is this model going to return confidence values? No, they want decisions. This is an inherent problem in AI with reductive logic for these tasks.
Likewise, dates. Is 17/08/22:
- 20170822 (Chinese write the year first, and in speech and chat they will say '17' not '2017'.)
- 20220817 (Everywhere But North America.)
Is 04/06/2009:
- 20090604 (Non-North American)
- 20090406 (American)
Without knowing the cultural context, the location, the document type, the document author, etc, you could be getting it wrong.