Cockroach Labs' AI 'hospital' shipped 1,238 PRs in five months
Five months treating bugs like patients and coding agents like a medical team
Cockroach Labs built MOLT Sinai, a system that treats bugs like patients and coding agents like a medical team. Over five months, it merged 1,238 PRs and discharged 1,322 issues, with only 7 reverts. The Db2 migration feature, which would have taken nine months and $160K manually, was completed in under two days for $4,172 in tokens. The system uses roles like Triage Nurse, Fellow, and Review Attending, with a human Chief of Medicine for escalations.
The token bill was $4172: 164x faster, and 38x cheaper.
- finnborge
This feels to me like a lovely example of both high-quality systems design and information theory. From my perspective, one of the more abstractly interesting thoughts surfaced by this write-up is the degree to which you've "encoded" a highly complex set of relationships through use of metaphor.
That type of encoding (starting with the statement: you're at a teaching hospital) is profoundly more efficient than having to deeply and reliably articulate what each of the various roles your agents embody are, let alone their interactions.
Perhaps there is more room for any of us to consider what existing systems or identities are described abundantly in training sets and can be leveraged to ritualistically encode these social/civic/cultural dynamics saying: "act as though you're ___." Not exactly a new point, but one this write-up certainly is pulling me towards.
Thank you for sharing and the care you put into writing this!
- traceroute66
> it’s consumed just over $135,000 in Claude tokens
Ironically also just like a US hospital. Go in and come out with a six-figure bill.
- anonymous908213
I can think of nothing I'd like to use less than a database or filesystem vibecoded by Gas Town-flavored psychosis. Roleplaying with LLMs is not the secret to producing amazing code.
- Selkirk
I really like this writeup.
I find the metaphor satisfying, though there's a reason metaphor receded into the bushes as a practice in XP. A great metaphor is great to have, but they don't always exist.
Fortunately with organizational systems, things will tend to rhyme. For example, a role that "Verifies that all processes have been followed before anything merges" seems like it could fit many different systems.
We're at this fun stage where people are just trying stuff and I love it. I hope we get more info from this experiment.
- james_marks
A hint at why GitHub actions have been unreliable. How many teams are running factories like this on GH infra now?
- ajstorm
Rafi and I, who authored this post, will be hanging out here for any questions people may have.
- gausswho
The next role: Insurance Rep.
Ensures all other agents are operating efficiently and within reasonable levels of token usage given the expected level of effort to 'resolve' the patient. In moderate to severe cases, may lead to patient defenestration or agent revolt.
Or perhaps, the author considered this role but found it typically costs more than it saves.
- K0balt
I set different kinds of structures for different projects,
Complete with setting and ambiance. It’s like agents work better if they are role-playing. It’s extremely disorienting and people with marginal mental stability are going to really have a bad time. What have we wrought?