Why Software Factories Fail: Harness Engineering Is Not Enough

I have mixed feelings about software factories!

On one hand, our core product is just simply not fit for them at its scale. We've tried but the project is large enough to require human input for every change. But we have AI automations for light code refactoring, writing tests, UI changes etc. and they work.

On the other hand, I started a number of small experiments to see how far software factories can be pushed and while the code produced so far is nothing spectacular I could easily imagine how this can be extended in the near future. Perhaps if you start from the ground up with the idea that the code will be written that way then you can come up with strategies and architectures that accommodate it. At least this is my thinking right now.

Anyway, it is all open source and documented here https://relentless.works/ I am not sure for long I will keep this running. I provide zero direction to where this is going. I have no idea what it will happen next. It is a fun experiment.

I have another such experiment with a trading agent. I thought it will loose all of the money in short time. For a while it was stuck with no open positions after it lost a bit. I decided not to intervene and just observe the behaviour. Recently it opened new positions which was an interesting development. It is still loosing money (~ -3%) but it has not lost all of them and given the current market circumstances I would say this ain't bad at all. It just shows that perhaps we might be a bit impatient wh […]

There's some good ideas and points in here, but this bit threw me:

> # We tried this

> In July 2025 we went full lights-off

Isn't it pretty well-accepted at this point that the models underwent a step-change in usefulness around fall 2025 / spring 2026? I know that I was able to start handing agents whole features after that, but not before.

I feel like any perspective/experience on "what agents can/can't do" from before that period is... maybe less than relevant to the modern era. TFA calls it out a few sections later with "But surely the models have gotten better since then", but then just writes off any improvement. That does not match my experience.

I find it amusing that people who are talking about Dark Software Factories, are talking about productivity in terms of number of pull requests or commits as a unit.

If we are going in the Dark Software Factory route, why aren't we calling the code units as bos (bunch of shit) yet.

Love the idea of RL for codebase health.

And a benchmark to measure against!

Imagine a "MaintainabilityBench" that rewards models which detect code duplication while working on a task and perform some refactor instead of glibly duplicating; or that detect the need for a new architectural layer, or that hoist a type constraint so there's no need for dumb casts. You can keep on imagining scenarios.

There are probably a few hundred distinct elements to RL for. The books "Working With Legacy Code" and "Architecture of Open Source Applications" would be great fodder.

Sadly don't have time to build it, there's this mountain of reviews in front of me...

To me, the thing that stands out about the whole state we're in here is PR review.

Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end.

I'm not sure how the best teams do PR review, from my perspective it sucks. I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed by pulling down the branch and opening the diff with $EDITOR.

These days I think there's really no excuse for the awful UX. Linear (a company that isn't even in the domain of code review) put out a basic PR review feature[0] that is already better than what GH offers. It's simple: point a small model at the PR, group file changes together based on theme, add some commentary and sort by importance (schema changes > openapi spec).

Immediately, so much mental load has been reduced without the reviewer or the requester doing anything. This feature is pretty damn basic, and I think there are obvious next steps like generating visualisations which a dedicated product could find the time to implement.

Keen to hear others thoughts on why this is the wrong approach, or if there are tools in wide use that solve for this, or why this isnt the right problem to focus on.

0 - https://linear.app/docs/diffs#guides

I think there is a fundamental issue here of what building software even means

If you think you can just assign Github tickets to AI agents and go drink daiquiris on the beach I think you'll find that you end up with more and more towers of abstraction and indirection. There are 'points of view' that emerge during coding I think. And at some point you as a human have to be like "wait... what if we use Redis here". "Wait.. the API is already returning the data we need". "Wait... let's not add customers to the report who have not been active in the past year". Stuff like that

> So, why can't models do software maintainability?

I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad design?" Of course the models aren't good at it ... they're not good at anything until you've tuned them and put them in a harness that rewards good edits and throws away (improves) bad edits. That doesn't explain why "models can't do software maintainability." The real question is why harnesses can't do software maintainability, and how to build a system that can do it. (I suppose that's the purpose of the ad at the bottom of the page.)