Stanford-led benchmark finds top AI agents solve only 30% of real scientific workflows
Terminal-Bench-Science: Evaluating AI agents on scientific research workflows
Terminal-Bench-Science, a new benchmark from Stanford University and the Terminal-Bench team, evaluates AI agents on 70 expert-curated tasks drawn from real research workflows across life, physical, Earth, mathematical, and engineering sciences. The strongest model, Claude Opus 5, resolves only 30% of tasks, while most models score below 10%. The benchmark is designed to be continuous, evolving with frontier AI, and is open for contributions from scientists worldwide.
Scientists, not model developers or data vendors, set the bar for scientific capability in AI.
- j_maffe
The tasks are the thing to really look at here:
https://github.com/harbor-framework/terminal-bench-science/t...
- johnnyApplePRNG
Not surprised to see Claude significantly higher in scientific intelligence than Sol.
You can tell that Claude really does grasp a wide array of highly specific scientific and mathematical nuances... where's codex is just basically for coding and that's it.
That's the feel I get from the both of them anyways and I've used both on the 20x plan for the past week at length.
Don't get me wrong, codex is great at finding bugs and building games. It's great.
- CJefferson
I worry this doesn’t check correctness. I’ve been finding Claude is lately awful at folllowing instructions, I’ll ask it to implement the algorithm from a paper and it will do something simpler and slower and when challenged do it’s stupid apology thing. It can’t be trusted with anything I’d put in a paper, it lies too much. 4.6 couldn’t do as complex tasks, but it would do what was asked.
- boorang
I got pretty good mileage out of context engineering, adding my personal coding heuristics to my AGENTS.md and referencing subdocuments on a "when doing X, consult Y" pattern. I assume others are doing similar things, but I was pretty surprised when I was able to get it to generate code that is pretty close to what I would do if I was doing it manually. I'm curious if scientists and mathematicians are doing things like that. "When I see X, I typically immediately check Y" or whatever their domain heuristics look like.
- jerpint
The fact that opus 5 is outperforming fable is odd to me
From personal experience, opus 5 feels net inferior to fable on almost every aspect (for coding tasks)