AI writes perfect code, but it's twice as sloppy as humans
If coding is solved, what now?: Measuring the sloppiness of code
LLMs generate almost perfectly correct code, yet they introduce unnecessary abstractions, duplicates, and bloat. Measuring this 'sloppiness' is hard: AI judges are unreliable, and human review doesn't scale. Simple metrics like lines of code, verbosity, and erosion reveal that agent code is roughly twice as verbose and eroded as human code. In iterative benchmarks, even state-of-the-art models fail to pass all tests, highlighting accumulating bad decisions.
The agent's code is on average roughly twice as verbose and eroded as human code.
- dang
All: please don't post generic reflexive reactions to titles. That's covered by this guideline, among others, in https://news.ycombinator.com/newsguidelines.html:
"Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead."
I've taken the provocative bit out of the title above, but please remember that we want reflective comments, not reflexive ones, in HN threads.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor....
- dherman
Really glad to see folks looking into quantitative approaches to give agents feedback on code quality. This post looks like a good start!
My main feedback for the authors would be, the most important problems for sloppiness are global properties, not local ones. In my experience an agent, like a human, has finite capacity for its attention, but if it runs into local sloppiness that gets in its way, it can fix it on a by-need basis. The technical debt issues that matter are usually global issues that aren't so easy to fix: they require global analysis and global refactoring.
I don't know the answer, but I think we're going to need ways to measure architectural properties, like separation of concerns, clear architectural layering, well-defined interfaces, etc.
- toddwprice
Coding is solved, perhaps, with unlimited token spend on a frontier model. It remains to be seen if it that is prohibitively expensive forever. At my company, we token maxed while the getting was good. But when we had to switch to Anthropic's enterprise plan, and start paying per token, the shit really hit the fan. Now we're retreating back to sane cost levels and finding that - guess what? - people power might just be more cost effective. AI of course is an immense tool to leverage, but still too expensive to create loops and let it run. This will change over time of course, but assuming it is a solved problem is nonsense. Maybe if we solve cold fusion, yes. Until then, evolution is winning the war on entropy.
- justinmarsan
Having reached the same conclusions as the author led me to create my first agent to do architecture review, and that's how I learned about the metrics behind good practices that I'd been following for years. LCOM, cyclomatic complexity, that kind of stuff...
It's so easy to ship a lot of code, more effort should be put into ensuring the code is correct, with self-improving feedback loops that involve developers, and dedicated tooling...
But again, a while ago, everything was about prompt engineering, and now you can express you idea vaguely and get a somewhat working result, so this likely will evolve fast as well...
- FiberBundle
Does anybody actually know whether there's a limit to the complexity LLMs are capable of dealing with in a codebase? It's very obvious that they don't write code that is suitable for people to understand it (and it's gonna get worse and worse the more RL is used to train these models), but if there isn't a point at which LLMs also struggle due to the complexity they introduce, then I'm not sure it really matters anymore for a large part of non safety-critical software. I really hope there is, because steering them is, I feel, one of the last competencies through which I can still add value, but is there actually evidence that these models struggle more with poorly maintained code?
- drsopp
Goodharts law is invoked here with no discussion. Are we sure that optimizing for minimum LOC that pass all necessary tests produces sloppy code? And: What is sloppy code anyway? If we could define it, could we throw the definition into context and tell the LLM to avoid it?
- conqrr
Coding is not just the program running in memory, its also the process of distributing the mental model of understanding among the team.
If humans increasingly are kept out of coding, then who holds the mental model?
If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM speak to technical decisions.
So is coding solved now? it was already solved decades ago.
- glouwbug
A good measure is communication. If there's a common understanding, then the origin doesn't necessarily matter.