Why Asking LLMs for Confidence Scores Is Completely Useless
Don't ask an LLM for a confidence score
I argue that asking Large Language Models to generate a confidence score for their responses is scientifically invalid and practically useless. While models show some latent internal awareness, they cannot reliably quantify their own correctness. Treating these self-reported numbers as trustworthy metrics is a psychological trick that flattens complex uncertainties into a single, misleading figure.
What you have built is a psychological safety trick. It makes the output feel more trustworthy without making it more trustworthy, and I'd argue the people shipping this experience are mostly lying to themselves about what they've shipped.
- SubiculumCode
In human memory, we often use a sense of familiarity to guide our memory decision, in the absence of explicit recollection of details. There is a whole memory literature about "recollection and familiarity" that dissociates the two cognitive processes, recollection which involves retrieval of specific details of an experience, and familiarity, which is a sense of memory strength, but absent of any qualitative detail. Familiarity is a faster process, and can often spur subsequent retrieval attempts that can lead to actual recollection..e.g. you see someone that seems familiar, but can't place where...and after a few moments, you remember who they were and where you had met them.
When measuring these processes, one approach has been to ask participants to provide confidence ratings. Recollection tends to lead to threshold-like, very high confident responses. Familiarity is more graded and continuous. Many then use a dual-process ROC model to identify the recollection and familiarity components, on average, of a person's memory of a memory test (see work by Andy Yonelinas).
This kind of work goes beyond memory, but applied to the general problem of how people judge their confidence in answers.
Its likely been applied to LLMs. A familiarity signal would probably be pretty easy to generate... The recollection kind of component might take some of those introspection type of approaches. For example, these papers, which I have not read, [1] https://arxiv.org/html/2603.17839v1 [2]http […]
- fhdkweig
That's what I thought was interesting about the Watson Jeopardy Challenge. Watson had a sense of its confidence, and it seemed to be accurate. The answers it got wrong were the ones it knew it had difficulty understanding.
3 videos at https://archive.org/details/Jeopardy_2011-02-14_The_IBM_Chal...
- bensyverson
I was designing a system that pulls structured data out of an unstructured transcript, and Opus initially suggested a confidence score for each value. When I asked it “do you think the model will generate a meaningful number there?” the answer was basically “ha, you caught me.”
What did work was asking the model to emit citations in the form of line numbers from the transcript. It dramatically dropped hallucination.
- ptx
Calculon: And you say you can guarantee me the Oscar?
Bender: I can guarantee anything you want.
- dmrivers
The statement near the top of the post
> "The short version: asking an LLM to generate a score for how confident it is in its own response is, from everything I can tell, completely useless."
is definitely too strong of a claim and directly undercut by what is said near the end of the post:
> "Tian et al. found in Just Ask for Calibration that with the right prompting strategy, RLHF’d models verbalize probabilities that are better calibrated than the model’s own conditional probabilities, and that prompting plus temperature scaling can cut expected calibration error by more than half. And Anthropic’s Language Models (Mostly) Know What They Know found encouraging results asking models to estimate the probability that their own proposed answer is true."
My own experience is that stated confidence is a helpful tool and of course you need a rubric and a proper prompt, but this is clearly less work than training a classifier (as advocated by the post) and requires less data.