AI's Chain-of-Thought Reasoning Isn't Always Faithful
Chain-of-Thought Reasoning in the Wild Is Not Always Faithful
Recent studies show that when faced with explicit biases in prompts, models often omit these biases from their Chain-of-Thought (CoT) output, revealing that verbalized reasoning can misrepresent how models reach conclusions. This work demonstrates that unfaithful CoT also occurs on naturally worded, non-adversarial prompts. When separately asked "Is X bigger than Y?" and "Is Y bigger than X?", models sometimes produce coherent but contradictory arguments, often due to implicit Yes/No biases. Rates reach up to 13% for production models, and even frontier models like DeepSeek R1 (0.37%) and Sonnet 3.7 with thinking (0.04%) are not entirely faithful. The study also identifies "Unfaithful Illogical Shortcuts," where models use subtly illogical reasoning to make speculative answers seem rigorous. Findings suggest CoT is useful but not a complete account of internal processes, warranting caution in agentic or safety-critical settings.
Our findings indicate that while CoT can be useful for assessing outputs, it is not a complete account of the internal process that produced the model's answer and should be used with caution in agentic or safety-critical settings.
- Planktonne
Of course not. Because the article uses the words 'thought' and 'reasoning' and even 'faithful' to mean something other than their normal meanings, but then expects them to behave exactly the same.
Every field has terms of art, and 'reasoning' is one for LLMs. But that doesn't mean it has the same properties as 'reasoning' in other contexts, because you're not referring to the same thing.
Why doesn't my asteroid belt buckle?
- florianherrengt
This paper puts words to something I’ve noticed repeatedly with LLMs, particularly Qwen3.6. When I read its reasoning, it appears to recognise the mistake and then carry on as if it hadn’t noticed it at all.
> models often determine their answers based on implicit biases tied to question templates, then construct reasoning chains to justify their predetermined conclusions
> its reasoning was correct right until the final step (Yes/No answer)
- bee_rider
I see this when just using some chat bot that shows the “reasoning” steps (ad-hoc observation of course, it’s really cool that people are actually studying it).
It is annoying when the bot seems be “reasoning” correctly and then makes an obvious mistake at the end. And perplexing when it seems to be completely wrong and then pull the right answer out of a magic hat at the end.
I guess it makes sense; the “reasoning” steps aren’t actually doing logic, just adding more context to influence the final generation, right? But it is weird to see.
- sergio_valencia
This paper made me wonder not whether the chain we can read is actual “thought,” but what conclusions we can draw by observing it. It is an output channel, but not direct access to the black box that creates it. The question pairs present an interesting experiment, but they also got me thinking about semantics: the same underlying relation can have many valid representations, and how models reason across those representations can tell us more about their stability and correctness. Basically, are models semantically consistent when given different representations of the same underlying relation? Do their answers transform as the relationship requires, and do their explanations remain consistent with that relationship?
- flyingpumba
First author here, happy to answer questions