Jepsen finds ParticleDB's "Full ACID" claim was premature, with 96 issues
Jepsen: ParticleDB 1.1.4
Jepsen tested ParticleDB versions 1.1.4 through 1.1.18.11.rc-22, the multi-model database Orkes built for AI workloads, and found 96 issues — crashes, deadlocks, data loss, partially-invisible rows, dropped tables reappearing in distorted forms, and strings turning into integers. Most required only a single server and client with no faults. Even at the strongest serializable setting, committed writes were still lost during normal operation.
By version 1.1.18.11.rc-22, all but ten of those issues appeared resolved. However, ParticleDB continued to lose committed writes during normal operation with the default occ transaction mode, and allowed internal anomalies and Lost Update with serializable, the strongest setting.
- the_duke
Brutal!
> ParticleDB 1.1.4 claimed to be a Postgres-compatible database ... Instead, we found a library of confusion
> exhibited frequent, profound transactional anomalies in essentially every way we could measure. We routinely observed the loss of committed writes, internal consistency violations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost Update). In the append workload, successfully appended values would routinely disappear from later reads of a given key. Our test suite inferred every kind of cyclic anomaly including G0, G1c, G-Single, and G2-item, but many of these can be attributed to more basic issues, like write loss. These issues occurred with a single server and without faults, and affected every transaction mode, interface, and WAL mode tested
- amluto
I’ve come to learn an interesting dichotomy in software:
Some software is correct by construction, in the sense that the overall design makes sense and the cases that don’t work are generally local bugs.
Some software is incorrect by construction, and it tends to work by some combination of luck and by carefully working around the incorrect design in most or all code paths that could expose the incorrectness. (AI can be quite effective at finding the relevant places to insert workarounds.)
ParticleDB sounds like the latter category. Plenty of large projects have portions that are in one category and portions that are in another.
As an aside, for the last day or two, I’ve been contemplating that the Linux kernel’s handling of folio pinning seems to be in the incorrect-by-construction category. Good times. No, I haven’t found a hard-to-fix bug that I would credibly characterize as a security or likely accidental corruption issue [0], but this seems mostly like luck.
[0] I do appear to have re-discovered a bug that some massive Anthropic dump from a couple months ago also found, and it’s getting fixed. If your application does “fixed” io_uring I/O or certain types of iommu-based or maybe RDMA on a memfd, you might plausibly encounter it on rare occasions. I didn’t bother making a reproducer.
- tensegrist
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available. It is unclear what properties might be proven, or whether they described ParticleDB as built.
- wofo
> Given the severity and extent of the safety errors we found, and the process which produced them, Jepsen recommends users halt further deployment of ParticleDB and migrate to other databases as soon as possible. Consider checking and/or rebuilding database state from more reliable sources, if available.
I guess that's what happens when you vibecode a database...
- edude03
> We present 96 issues in ParticleDB, ranging from the pedestrian (crashes, deadlocks, data loss, etc.) to the piquant (partially-invisible rows which could not be deleted, WHERE 4 = 5, dropped tables re-emerging in distorted forms, confused columns, strings turning into integers, and so on).
Definitely seems worse than usual (looking back most databases seem to have <10 issues total, and the majority seem "mild") - since it's a database for AI was it also likely vibe-coded or just new? Then again even if new, I wouldn't expect so many issues in a stable release
- mxey
It’s wild how bad it is. Not surprising for a vibecoded database, but why did they hire Kyle for a test, instead of first doing ANY testing themselves?
- eqvinox
> Strings sometimes turned into eight-bit integers (#37)
FINALLY we know what happened with eMudhra's keys!
(Sorry. Really couldn't resist.)
- t3ntxcles
Shipping a bug like "WHERE 4 = 5" returning rows says it all to me