Calibration, Not Speed, Is the Real Bottleneck for Production Classifiers
Jev and System One Models: Calibration Beats Accuracy
TypeSafe AI's Jev, a non-autoregressive System One model, promises calibrated probabilities in a single forward pass. Drawing on his COMPSAC paper where a Random Forest hit F1 0.958 but a majority baseline hit 0.957, the author argues calibration is what limits production classifiers. He outlines an experiment comparing Jev, a calibrated forest, and an LLM on PR acceptance, measuring Brier score and expected calibration error.
A confidently wrong answer in a valid schema is still a wrong answer.
- rahimnathwani
The way he proposed to deal with "How urgent is this customer support message?" is "a score over ordered levels".
I'm not sure that's a good way to use Jev for this use case. If I had this problem, I would ask Jev to answer several different yes/no questions about each message, and then use the probabilities as inputs into a logistic regression model that predicts urgency.
If you ask the model specific yes/no questions which can be answered reasonably objectively from the input, I think the answers are going to be more stable over successive generations of models.
e.g. if you ask 'Is the customer angry?' I'd expect that answer to have high agreement between models and between models and humans. But directly answering the 'is it urgent' question is much harder. (Although I suppose you can try to put the rules in the prompt.)
- amluto
It seems to me that Jev cannot be usefully calibrated out of the box, in a fairly strict sense. Suppose I have a pull request and the state is the title, description, etc. The question is “Will it be merged?” (it doesn’t really matter whether it’s a choice or a “noul” [0]).
Now consider that two different projects may have radically different criteria for accepting a PR. And the two projects may have different probabilities for acceptance of a random PR from the distribution of PRs they get (i.e. the overall fraction of PRs that are accepted). So what could Jev possibly return that is “calibrated” for both? It doesn’t even have an “I have no idea” option because the output schema cannot distinguish between “I am confident that there is a 50% probability that the answer is yet conditioned on the state” and “there is no useful information contained in the state that I can extract and therefore you should assume that you posterior distribution is the same as your prior”.
For fun, I gave Jev some irrelevant state and asked it various questions for which the state was useless (I picked sporting outcomes), and it was 0-for-3 at giving yes/no probabilities that were particularly close to the obviously correct no-information answers or close to 0.5 in cases where the prior was far from 0.5.
This is a silly test, but I’ve personally encountered genuine production situations where the best classifier available (or at least the best one available at any cost remotely close to what i […]
- svg7
Calibration has to be measured on your dataset. Just because the probs sum to 1, does not make Jev or Jev-like models claibrated. For folks interested in digging deeper into calibration, studying ad click prediction models (where calibration is super important) is a good place to start.