Jev, TypeSafe's New Classifier, Is Barely Better Than Random at Probability
How accurately calibrated is Jev?

TypeSafe's Jev, a 'System One' decision model, outputs probability distributions over multiple-choice answers. But when tested on 1,000 physics problems with known distributions like Maxwell-Boltzmann and Gaussian, Jev scored a mean total variation of 0.518—barely better than the 0.546 from a uniform guess. It tends to be overconfident and peaky, often copying parameters from the prompt. The author questions using Jev as an automated judge for LLM answers.
If Jev were to completely punt on the answer, and spread the probability evenly over all possible choices, it would score a mean TV of 0.546. But Jev scores 0.518.
- clarle
Wouldn't asking humans have this same kind of problem too?
If you asked a set of humans to generate a random distribution of heads or tails from coin flips, it wouldn't be similar to a real world distribution of coin flips either because we also have our own biases. [1]
[1] "Heads or tails?"--a reachability bias in binary choice" - https://pubmed.ncbi.nlm.nih.gov/24773285/
- amluto
I’ve been contemplating this situation. I think that what the author wants out of a Jev-like model is not at all what I want out of it.
> I decided to check this on questions where the answer is well understood. For example:
> A classical particle of mass m is embedded in a system at thermodynamic equilibrium with temperature T. What is its velocity v?
If I feed that into a model, the answer I want is: “the combination of the model and the provided state has nothing useful to add to your prior”.
If I want to know the Maxwell-Boltzmann distribution, I can look it up or I can derive it or I can ask a fancy LLM to do it for me (at the cost of some reasoning tokens and some time - unless I’m using an ultraspeed inference system, I’m not getting this answer in 50ms). [0]
Similarly, if I want to know that 73% of incoming customer support requests are spam/fraud, I should measure that - it’s a property of my system, it takes some manual classification and a database query, and it will be a different percentage than your customer support system would see. I neither expect nor want my classifier to know this (unless I’m using a conventional classifier manually trained on my data, and the whole point of Jev is to avoid this).
What I want out of a system like Jev is to tell me how the probabilities change as a result of the per-sample data I provide. Which, is the case of this Boltzmann distribution question, is nothing: I provided no data and the classifier can infer nothing.
[0] A re […]
- dvt
> So Jev does actually know which distributions are correct, it just fails to produce them
Claims like this needs to be deeply analyized. Models do have some emergent capabilities[1], and I think there's a lot of evidence to show that semantics is actually learned (Word2Vec), and some math seems like it also might be learned (e.g. modular arithmetic). But it's hard to exactly say where there's some internal mechanism generating a true answer and where we're just getting lucky with some distribution so the answer just seems right.