We're Normalizing Inexplicable Software Failures, and That's a Tragedy
The Normalization of Inexplicable Failures
The author argues that LLM-accelerated development is making software failures more common, but the real danger is that we're starting to accept "stupid thing sucks" as a final answer. Using the example of Jev, an AI model that returns typed values with confidence scores, the post shows how builders ship opaque systems without evals or accountability, and users are left to discover failure rates themselves. The tragedy is that we're engineering systems where neither user nor builder cares to check if there's a body behind the door.
The tragedy of software engineering today is that we are actively engineering systems where neither the user nor the builder seems to have any interest in checking whether or not there's a body behind the door. We just shrug and conclude: stupid thing sucks.
- pmarreck
I am big on reproducibility (nix aficionado) and determinism (flagging test failures are a red-alert, all-hands-on-deck situation in my world) and correctness.
I am also big on testing (the correct things). And nine-nines (big on Elixir).
And... I'm also big on agent-assisted dev. Which requires pretty much every check in the book to stay productive in. And that's fine to me. I've seen bugs that I wouldn't have made myself. And I've also seen my own bugs fixed. They've all gotten fixed in short order. I don't see why this is a problem.
Raise your personal standards.
Thing is, the unreliable-software situation was already untenable before agents (in poor hands) made it worse.
- adamddev1
Excellent post. People always defend agentic/LLM-driven development by saying, "Well it's good enough", or "It works most of the time."
That may be tolerable for some user-facing app. But what if we start normalizing failures in the libraries, the infrastructure, and the compilers? Everything descends into a mess of unreliability, and that slows EVERYTHING and EVERYONE down.
- theamk
> When a button breaks on a website, I have a model about what should have happened. Somewhere a contract got broken. [...] I might not have access to debug just an HTTP status 500, but I expect there to be somebody whose job is to understand why the endpoint is 500ing. The ownership is well-defined albeit opaque³.
> For many users, however, the actual experience is roughly just "stupid thing sucks." Software already feels capricious; more failures just change the rate of frustration.
I am betting author does not use cloud services much. It is not just "users", it's developers as well. Github is returning 5xx? AWS service does not work? Your email did not get delivered? Nothing we (developers) can do, "stupid thing sucks".
- layer8
> This leads to a normalization of inexplicability.
It’s also tightly connected to a normalization of lack of accountability.
> This isn't "getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" -- you still have to do the hard part.
This is probably losing the younger portion of the audience by now. ;)
- WorldMaker
"Confidence scores" have always implied an anthopocentric meaning that doesn't exist. An algorithm doesn't have "confidence" in the way that a person has confidence, but as soon you put something with that name in front of a business person they assume the number is always a meaningful "letter grade curve" or "universal percentage". I still believe so much that the old quote to "there's lies, damned lies, and then statistics" remains a key to understanding so much why ML is leading to dumb outcomes versus hype. People don't understand statistics, so machines that produce nothing but statistics especially confuse people. (I feel this applies to LLMs as well.)
- teraflop
The "normalization of inexplicability" is indeed infuriating. It has always been bad when it comes to computer software, and it's increasingly creeping into other consumer products that depend on embedded software.
I bought a new electric car recently. For the most part I've been quite happy with it. Shortly after I bought it, it started popping up a warning message saying "check EV system" every time I started it. By the time I brought it into the dealership, the warning had gone away, and the technician just told me something to the effect of "eh, I guess it just does that sometimes, let us know if it happens again." Hardware fault? Software bug? Who can say?
Like most modern cars, it has connectivity and Google Maps built into the infotainment system. The vast majority of the time, it works fine. Sometimes it says it has no connectivity (meaning no traffic data and suboptimal routes) for the duration of a drive, even in areas with a strong cell signal where it normally works fine. Sometimes the car says it has connectivity, but Google Maps still thinks it's offline. Sometimes Maps will actually load and display a route, but the "start navigation" button just spins forever as though it's still waiting for something. Are these related issues? Is there a common cause that might be fixable? Who can say?
(Conveniently enough, the warranty specifically does not cover any failures of software or firmware to operate correctly.)
- nizarmah
I've seen people opt to refactor their entire codebase from one lang to another just because AI made it so much easier. Sure there were problems before, but now the new code with the better language is not readable and needs another refactor once this entire thing is done.
- benjaminsky2
I’ve validated Jev’s confidence score. Accuracy scales linearly with confidence for the 3 use cases I tested. >.9 it matched a human labeler. I immediately discovered a user behavior I didn’t expect for ~$3. I can now mitigate in real-time due to low cost and latency. This may have a major positive financial impact for all our customers.
Not sure why anyone would feel the need to dunk on this thing without showing a real failure example.