Jev's Calibrated Probabilities Are a Myth, Says This Analysis
Jev Can't Be Calibrated
Jev, TypeSafe's System One Model, returns typed decisions with probabilities, but its calibration claim doesn't hold. The author argues that calibration depends on your data distribution, not just the model, so Jev's probabilities can't be calibrated for every user. Even worse, Jev reportedly assigns a 0.92 probability to a fair coin landing heads when the true probability is given in the prompt. The advice: treat Jev's outputs as scores, not probabilities, and recalibrate on your own data if thresholds or expected costs matter.
While writing this post I found these tweets, where Jev says a fair coin lands heads with probability 0.92. That is worse than the drift explained above. The true probability is in the prompt, and the model still does not report it.
- kantahayashi
I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%. I also tested with a fair coin 200 times and got 0.92 probability.
I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.
Write-up: "Jev Does Not Play Dice"
https://kantahayashiai.github.io/posts/jev-does-not-play-dic...
- jackb4040
This is why I don't understand why everyone's freaking out about it. By far the biggest problem with LLM classifiers is that they treat every individual business as the blurry average of all businesses in their training data. Being lighter is fine if you control for everything else, but at least at my company we would actually have room for a significantly more expensive / slower classifier if it were demonstrably better at following instructions.
- abhgh
I like this post. I haven't had time to dig into Jev (they aren't accepting new signups), but calibrated probabilities is one of their pitches that caught my attention. And I was wondering how does one offer them on user data. Standard calibration essentially ensures that if a score of 0.8 accompanies a positive prediction (assuming the simple case of binary classification), then if you gathered together all predictions with a score of 0.8, around 80% will be correct.
If you have just one example you're sending to a model, how would they guarantee 80% over your data?
FYI, for an overview, scikit's page on calibration is great [1], and my answer on Quora from a long time ago covers a specific type [2].
[1] https://scikit-learn.org/stable/modules/calibration.html
[2] https://www.quora.com/How-is-isotonic-regression-used-in-pra...
- bnbn88
This hype is caused by the price and the speed since most people don't know about small fast models and use big models for everything.
- time0ut
I have been running a series of experiments on Jev since its release targeted at understanding it, seeing how it handles real use cases I have, and maybe figure out what it is inside.
Some of my tests do point towards what this post says. I was not successful in getting it's score to align with an existing rubric I had. It 'worked' but it was off and compressed from where I wanted it to be. Not a bad starting point, but I couldn't get it to move to where I intended the rubric to be. It wasn't the most robust test and I didn't spend a lot of time trying, but it wasn't just instantly magical.
However, it does seem genuinely useful just by being fast and cheap and good enough, so I am still a bit hyped.