Benchmarking Opus 5 on SlopCodeBench: Why AI Still Needs Human Steering
I tested Opus 5, Sonnet 5, and Opus 4.8 on the new SlopCodeBench, which simulates real-world software evolution with hidden requirements. While Opus 5 achieved a 24% strict pass rate, it generated five times more code than older models and still failed to complete challenges without defects. The results suggest that current models cannot yet run lights-off for complex engineering tasks, as every dollar spent bought correctness but nobody bought enough of it.
every dollar bought correctness. nobody bought enough of it.
- robbomacrae
SCB is definitely an underrated benchmark. For me the unique selling point is that it more closely mirrors software development by not stopping after a single task. The agent has to keep code clean. The only disadvantage is all the problems are greenfield and not git inited so the agents don’t make use of git diffs.
I’ve used SCB as part of my assessment of agent skills (superpowers, GSD etc) https://orcabot.com/labs/do-skills-improve-coding-agent-accu...
There is a small but growing community on discord for discussing SCB so if interested please join https://discord.gg/BrC4BA9sVj
- WilcoKruijer
I hope the big labs will start using this benchmark in their RL pipelines. Reducing complexity in generated code should be the number 1 priority, in my opinion. The holy grail for me is models implementing features while reducing LoC (i.e., choosing the right abstractions).
What is also nice about this benchmark is that it can be used to iterate on prompts/skills for reducing code complexity.
- jrflo
The author of the original paper did not include error bars on the cost and quality results, which is a bit alarming. They did include a +/-, but did not list explicitly if that means standard deviation, range, or 95% CI, etc. Another table is listed as that being one standard deviation. If we take those results to be a standard deviation as well, I think we can conclude that there is no statistical difference between the outcomes of all models in terms of quality and cost, as the error bars are all overlapping.
- dmrivers
I always have Claude recite a pledge before starting coding to fix redundant code it notices over time. It does seem to find redundancies, but only when I point out bugs, that's when it goes into fixing mode and actually applies my Don't Repeat Yourself preference from the CLAUDE.md.
The original paper cited by this post does try to see if improved prompting will make a big difference in the end using a `plan_first` prompt variant, but find no influence on pass rate at the end of the benchmark. The `plan_first` seems to assume coding agents will just refactor once they finish features, but I don't think they tend to refactor significantly unless they are told to fix bugs rather than build features. The benchmark leaves tests hidden with no fail-to-pass feedback, so that may be why degradation is monotonic.
- Vgoose
Nice! I actually ran across this paper+benchmark recently, too. It's the first I've found that start to aim at some of the non-functional and longitudinal requirements that I think have always been an important part of writing production code.
It's especially relevant now that models are good enough to solve ~most point-in-time problems.
Some relevant but disconnected thoughts:
- deterministic scores are so nice
- what "maintainable" is is probably some high dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is
- another signal I've been thinking about and I'm seeing increasingly get brought up is the state space of a system; I'm seeing formal methods pop up a lot recently
- sothatsit
This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt.
I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.
- mellosouls
This is nice, but would be really useful measured against human performance, and yes - I understand that is likely a difficult challenge.
Many people though are going to read the headline figures and think it means - say - Opus 5 is only a quarter the strength of a human coder.
- Johnny_Bonk
I haven't joined your chats in a while but glad to see you put this together, I truly feel as though opus 5 is not much of an improvement. The only time i ever felt a wow factor was opus 4, 4.6 and fable pre trump admin lobotimizing