OpenAI's newest models stumble on SimpleBench, scoring below a high school human baseline
Astra 6.2 and Sol 6.2 flop at Simple Bench
SimpleBench tests everyday reasoning with over 200 multiple-choice questions on spatio-temporal logic, social intelligence, and trick questions. A non-specialized human baseline of 83.7% still beats most models, and OpenAI's new GPT-6.2 Astra and Sol actually regressed, landing at 77.3% and 69.9%. Anthropic's Claude Fable 5.2 leads at 94.9%, with Google's Gemini 4 variants close behind.
The results of SimpleBench confirm that the memorized knowledge, and approximate reasoning retrieval, utilized by frontier LLMs is not always enough to answer basic questions just yet.
- 0xbadc0de5
Having used both Sol-6.2 and Astra-6.2 heavily over the last few days (as well as regularly using Claude Opus and Fable) they are marked improvements over the previous releases and comparable to Opus-5.5 and Fable-5.2.
I've stopped trusting any of these influencer-benchmaxxers because they generally don't run any of the workflows of interest to me - it's completely synthetic. The only benchmark that matters is how well any model works for your current use-case.
- nsingh2
The 6.2 models are relatively meticulous and thorough. Great code reviewers or at mathematics.
However they seem to be relatively worse at deriving intention, or dealing with vague and underspecified tasks in a way a human would find reasonable. I've seen some people jokingly refer to them as "autistic".
All GPT models from GPT-5 onwards seem to have this kind of style to them, some worse than others.
- fullstackwife
I have an idea for a benchmark: train a model on a knowledge cutoff lets say 1945, and simulate world development for the next 70 years. Then see if it invents things human did, and avoid our mistakes/dead ends too.
- ricardobeat
Neon / Carbon / Argon / Hydrogen are all in the same ballpark, why did they even release four different models?
Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering.
- algoth1
Phillip has updated the leaderboard and surprisingly every new model release got substantially better, except for the OpenAi models. 6.2 Sol is 30th whereas 6.1 Sol is the 10th. Astra 6 is 9th, Astra 6.2 is 15th. By comparison Fable 5.2 became the 1st and Fable 5.1 is 7th. All the new google models, even hydrogen, are above the human baseline. By contrast, Astra 6 pro is the only openai model slightly above the human baseline at 8th place
- martinjc
For my experience, i feel it follows instructions too well. Grey areas are completely cut out, and sure thats good sometimes. But for my usecase, i liked when the response felt smart and ready to ask questions.
- davidmurdoch
I've been finding 6.2 Sol doesn't act like an architect and doesn't doubt its instructions or existing code at all. It's back to "you're right to push back" responses when questioned about its plans/reasoning in the slightest.
- hoppp
All the new Gemini models perform better on this than Opus 5.5
Is that something that can be noticed during real life use?
I really like opus 5.5, It's perfect and I haven't tried the new gemini models.