The Real Bottleneck for Self-Driving Codebases Isn't the AI—It's the Dev Environment
Towards Self-Driving Codebases
Agents can one-shot games and execute complex migrations, yet human engineers still drive most real software work. The author argues that the limiting factor for dev agents is not the models but the environments they operate in. To reach self-driving codebases, we need agent-legible dev stacks, global memory across tools, and codebase rot prevention. The path forward: mine codebases for bugs, fix them, and use the traces to prioritize agent-readiness investments.
At this point the limiting factor for dev agents is the environment they operate in, not the models and harnesses themselves.
- CharlieDigital
> It’s actually fine if agents make a lot of boneheaded mistakes. What’s not ok is if they keep making the same mistakes.
I worked in life sciences for a bit. There is a process in clinical trials called corrective and preventative actions (CAPA). You'll also find this in other areas where failure tolerance is low (e.g. aircraft).
It's simple: when a mistake happens, you run your CAPA process (Google CAPA form and see examples to extrapolate what that process might look like) and determine the root cause and the correction to the process that allowed the mistake to happen in the first place.
(At least as a SaaS vendor in life sciences, when we had a CAPA (e.g. after a SEV0 failure), it would be folded into our SOPs and then we would be required to retrain on the SOP. Auditors would want to see our evidence of CAPAs, the versions of our SOPs, the records of training. All to extreme for most shops, but I add this for context/color)
This is something most eng shops do not have the discipline for since it requires some diligence.
Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
- calldacopsidc
Sorry to get hung up on what is probably just flavor text, but the first sentence.. "Agents can oneshot games that are actually fun", uhhh such as?
Would love to be proven wrong here, feels like I've just seen the same basic one-level kart racer in like 3 model announcements. Not even sure if they were one-shotted, I certainly would not describe them as "fun"
- 2001zhaozhao
In the end, we already have processes that work for humans and we know the types of testing, verification and review that makes a codebase grow healthily. We just need to adapt these designs the best we can to AIs.
AI have a lot of advantages over humans like speed, controllability, good defaults (e.g. Claude will write helpful comments unprompted which is better than most engineers), and the ability to implement best practices like comprehensive testing with much lower cost than a human team. Using AI in my personal projects, I am able to write software at a much lower bug rate than manually, because now I can make an automated test suite whereas previously I had no time to do so. However, AI also has downsides like rigidity, limited context window, and lack of creativity by default which needs to be resolved with harnessing (e.g. different agents with separate, well-compressed contexts) or worked around with human review layers.
I suspect the best core pattern for coding automation will be to create an engineering manager agent layer that can flexibly apply a set of engineering practices to the concrete problems at hand. It would handle all coding agent pings before it reaches a human, with prompting so that it reliably considers the big picture when making decisions. Very rarely, it would escalate an issue or plan review directly to humans when it is truly blocking, but most likely it would organize the information it obtains from the lower layer coding agents and surface […]
- aerodexis
Makes sense to me - I see it going down two separate routes in parallel:
One is to take the human out the OODA loop for cyber defense. Servlet libs (for example) are going to become fluid, self-modifying things w/ contracts that operate much differenlty from how they do today. The engineering practice around these things will need to change.
The second is that UX will be self-modifying. Just like how pi can modify itself, I can see this being a general practice for user-facing applications. Perhaps a text-box in which users can describe tweaks in can request changes to how the application functions. Engineering an application will focus on modeling the non-negotiables of how an application works, and providing the correct primitives for user-driven LLMs to modify software on the fly, as well as track and rollback changes. There'll also be funsies around how to ensure that ads get delivered regardless of the user trying to get rid of them.
- jakeinspace
Kind of feels like we're in a similar place with AI programming as we are with self driving vehicles. Good enough for the typical case, maybe even better than most humans, but bad when dealing with novel situations or situations which require a really cohesive model of the system (code, business requirements, physical world, whatever).