PostHog's Jeeves beats Jev on JevBench by thinking before deciding
Jeeves. Reasoning improves Jev-like decision models
Jeeves is a 9B Jev-style decision model from PostHog that reasons before answering yes/no, multiple-choice, and rating questions. Trained with SFT and CISPO on Qwen3.5-9B, it hits 0.889 on held-out test data versus Jev's 0.857, and 0.935 on JevBench's public tiers versus 0.866. A block-4 diffusion drafter speeds up reasoning chains by 1.6x, with full code, data, and a drop-in Jev-compatible SDK.
A 9B Jev-like model (Qwen3.5-9B, LoRA, pointer head) that thinks before it decides, with a block-4 diffusion drafter and the full training code and train/dev/test data.
- HarHarVeryFunny
The number of people who feel the need to try and argue that you don't need Jev can only be astroturfing by those with something to lose - Anthropic and OpenAI employees.
Like it or not, companies are going to use Jev unless you can offer something just as cheap and fast.
I wonder just how much of the business automation market, previously held by LLMs, is at risk here?
- sharih
What is the point of this, if it is p90 17 seconds? Might as well use an LLM. The beauty of Jev is that it is dirt cheap and insanely fast.
- TN1ck
I just did a run with a benchmark I just used to test other models against. (It's about detecting irony in german soccer tweets). On my M5 Pro with 48GB it took over 30min to decide on just 100 tweets, the thinking definitely takes long.
It performed quite below Jev, but above other open decision models I tested (68 correct vs 79 correct for Jev - see [1]). I'm running it for the moderation benchmark as well, but that will probably take a few hours on my machine.
- itzikkatz
Cool engineering, but 17s p90 latency kind of defeats the point of a Jev-class model, which is supposed to be fast and cheap. Losing 10 points on MMLU along the way doesn't help.
- thm
Ask Jeeves - Only took us 30 years to come full circle.