Philip O'Toole Built Distributed SQLite in 2014, Eight Years Before Anyone Sold It
Twelve Years of Distributed SQLite
In September 2014, Philip O'Toole combined SQLite with a Raft log and released rqlite, likely the first open-source system to do so. He traces twelve years of the category: Canonical's dqlite, Ben Johnson's Litestream, cr-sqlite's CRDTs, the libSQL fork that became Turso and was acquired by Supabase, and Cloudflare's D1. The design space split into consensus, streaming, and merge approaches, while the venture money went to managed platforms.
But it is worth recording that the idea was open source for eight years before anyone sold it.
- piterrro
Having used D1 in production - it's not very reliable. There was a comment on HN from one of D1 devs [1] saying that D1 is a sql wrapper on top of durable objects that mimicks sqlite distributed like nature.
> I'll let you in on a sort of dirty secret:
It's almost always better to use Durable Objects storage, rather than D1. Even if you only want a single global database, it's better to implement that as a singleton Durable Object, than by using D1. Because that's all D1 itself actually is: a singleton Durable Object that exposes an API to its SQLite database. It's just a wrapper.
- AgentME
The comparisons between various projects, including between rqlite and Litestream, were interesting. (rqlite uses a consensus algorithm between nodes to keep them synced and allow changing the primary node, while Litestream just syncs itself to S3 so another node could take over with the data from there.)
I fully expected the article to get into D1 shortly and mention how that's about the same relationship between Cloudflare Durable Objects (which D1 is based on) and celld, an open source reimplementation of Cloudflare Durable Objects which writes to S3. But then the article treats D1 with a very sudden snarky disinterest, and then the final section feels awkwardly defensive of rqlite. It feels like an AI response to a different prompt than the rest of the article. (It seems to me (and Pangram) that the whole article is AI written but I found it decent quality until those last two sections.)
- mmnfrdmcx
As someone who loved sqlite and the simplicity, the truth is that with agents I am finding it is extremely easy to manage postgres and it is realyl nice having all the SQL features of postgres -- sqlite is quite limiting in a lot of SQL. So if you are considering remote sqlite and all that, feels a better time than ever to just use a database server.
Of course, sqlite is amazing for local tools etc.
- vikrantrathore
Its funny, when I was reading part time parliament paper (paxos) and while learning zig and odin language created a pure paxos library and then the first application for that library was to build a multi-master sqlite. The two approaches I took are very different in case of zig I build zaxonlite (https://github.com/insanai/zaxonlite) and have a benchmark on it that includes rqlite. It uses WAL replication as strategy. The second attempt I tried in odin is sqlodin (https://github.com/insanai/sqlodin), in this the system does not use WAL but run sql query on each node, as result subset of sql query in sqlite are not supported.
- Kinrany
Is there a project that does all of these together?
- embeddable like sqlite
- clustered like dqlite
- has a standalone node like rqlite
- can be backed up to S3 like litesteeam
- optionally AP like cr-sqlite alongside CP