An LLM's confidence score is just text, not a measurement
Using LLM for confidence scoring is a bad idea
Ask an LLM how confident it is and it will always hand back a precise-looking number — but that number is simply the most plausible next token, not a reading from any internal gauge. Rerun the same question and it shifts; reword it and it shifts again. A trustworthy confidence score needs two things an LLM never provides: it must reflect how much evidence went into the prediction, and it must be calibrated against real outcomes. The fix is to build a rule from observable signals and grade it against a record of what actually happened.
The model writes it, token by token, in the same way that it writes any other text. "0.85" is a plausible continuation of the conversation. It is not a measurement of anything.
- madrox
I got a degree in statistics, and what I remember from those times is that we spent maybe 10% of our time learning how to build the model and 90% of the time correctly interpreting the model output. Of course, in the time I learned statistics it was almost exclusively applied in hard science, finance, and actuary. The wild west of machine learning hadn't gotten started. I wonder if they teach it differently now.
When we start using big datasets where n went into the tens of millions we kinda stopped bothering because margins were so small it was effectively certain you found the mean of some range. However I think in that time we forget what confidence scoring actually is and how quickly multiplied alpha can balloon on you.
- luisfominaya
I looked at how consistent LLMs are by asking them to forecast events from Polymarket with slight variations in the input, such as negating the question.
You can see for yourself the results here: https://noisebench.com/
- Ozzie_osman
We've found that you can fine-tune confidence scores for your domain into existing models. I think the way this works is that you basically help the model understand what types of inputs it tends to get right and which ones it tends to get wrong. This seems like a reasonable thing you'd expect a model to be able to learn
- orbital-decay
Of course an LLM does have a latent concept of its own confidence that is internally consistent, and is perfectly able to articulate it, the author just chose the worst possible way to ask. Using a 0-100 scale for anything is a bad idea with both LLMs and people. Instead of rating it 0-100, split your problem into many dimensions/axes and give each a small and meaningful enum or just a less granular numerical scale (ideally just a binary choice). Then apply weighted scoring to the result. You'll be surprised how much better it becomes at meaningfully expressing its confidence this way.
YouTube removed the 1-5 scale for the same reason, people have no idea what to do with it.
- visarga
In my own words - you can't calibrate confidence of a model without a calibration set. Confidence is relative to some labeled dataset, it is established on a set of buckets or intervals, and is population based, something you can't check for a single prediction. Assign per case, verified per population.
I thought the way Jev was presented was sneaky - you can't say "we calibrated the model for you" and skip having a task. specific calibration set. They couldn't have known all our future tasks.